CLOTHO: Directed Test Generation for Weakly Consistent Database Systems

TL;DR

CLOTHO is a novel testing framework that automatically detects serializability violations in Java database applications running on weakly consistent storage, improving debugging and reliability in distributed systems.

Contribution

It introduces CLOTHO, the first automated test generation tool combining static analysis and model checking for serializability violations in geo-replicated databases.

Findings

Successfully detects serializability violations in benchmarks

Generates concrete test inputs for real deployment environments

Enhances debugging of weakly consistent database applications

Abstract

Relational database applications are notoriously difficult to test and debug. Concurrent execution of database transactions may violate complex structural invariants that constraint how changes to the contents of one (shared) table affect the contents of another. Simplifying the underlying concurrency model is one way to ameliorate the difficulty of understanding how concurrent accesses and updates can affect database state with respect to these sophisticated properties. Enforcing serializable execution of all transactions achieves this simplification, but it comes at a significant price in performance, especially at scale, where database state is often replicated to improve latency and availability. To address these challenges, this paper presents a novel testing framework for detecting serializability violations in (SQL) database-backed Java applications executing on weakly-consistent storage systems. We manifest our approach in a tool named CLOTHO, that combines a static analyzer and a model checker to generate abstract executions, discover serializability violations in these executions, and translate them back into concrete test inputs suitable for deployment in a test environment. To the best of our knowledge, CLOTHO is the first automated test generation facility for identifying serializability anomalies of Java applications intended to operate in geo-replicated distributed environments. An experimental evaluation on a set of industry-standard benchmarks demonstrates the utility of our approach.

Tables

Table 1. Table 1 . Consistency and isolation guarantees

Guarantee	Specification
Causal Visibility	${\Psi }_{\text{cv}}\equiv \forall {\eta }_1{\eta }_2{\eta }_3.\mathrm{𝚟𝚒𝚜}({\eta }_1,{\eta }_2)\wedge \mathrm{𝚟𝚒𝚜}({\eta }_2,{\eta }_3)\Rightarrow \mathrm{𝚟𝚒𝚜}({\eta }_1,{\eta }_3)$
Causal Consistency	${\Psi }_{\text{cc}}\equiv \forall {\eta }_1{\eta }_2.{\Psi }_{\text{cv}}\wedge \left(\mathrm{𝚜𝚝}({\eta }_1,{\eta }_2)\Rightarrow \mathrm{𝚟𝚒𝚜}({\eta }_1,{\eta }_2)\vee \mathrm{𝚟𝚒𝚜}({\eta }_2,{\eta }_1)\right)$
Read Committed	${\Psi }_{\text{rc}}\equiv \forall {\eta }_1{\eta }_2{\eta }_3.\mathrm{𝚜𝚝}({\eta }_1,{\eta }_2)\wedge \mathrm{𝚟𝚒𝚜}({\eta }_1,{\eta }_3)\Rightarrow \mathrm{𝚟𝚒𝚜}({\eta }_2,{\eta }_3)$
Repeatable Read	${\Psi }_{\text{rr}}\equiv \forall {\eta }_1{\eta }_2{\eta }_3.\mathrm{𝚜𝚝}({\eta }_1,{\eta }_2)\wedge \mathrm{𝚟𝚒𝚜}({\eta }_3,{\eta }_1)\Rightarrow \mathrm{𝚟𝚒𝚜}({\eta }_3,{\eta }_2)$
Linearizable	${\Psi }_{\text{lin}}\equiv \mathrm{𝚊𝚛}\subseteq \mathrm{𝚟𝚒𝚜}$
Strictly Serial	${\Psi }_{\text{ser}}\equiv {\Psi }_{\text{rc}}\wedge {\Psi }_{\text{rr}}\wedge {\Psi }_{\text{lin}}$

Table 2. Table 2 . Serializability anomalies found and manifested in OLTP benchmark suite.

Benchmark	#Txns	#Tables	Max. Length	#Anmls	#Dist. Strcts	Avg Anlys	Total Anlys	Auto Replay	Man. Replay
SEATS	6	10	4	32	18	26s	1970s	26	6
TATP	7	4	15	0	0	-	55s	-	-
TPC-C	5	9	4	22	13	118s	3270s	18	4
SmallBank	6	3	4	60	15	3s	264s	50	10
Voter	1	3	20	0	0	-	8s	-	-
Twitter	5	5	4	2	2	19s	211s	2	0
Wikipedia	5	12	4	3	3	227s	4343s	2	1

Table 3. Table 3 . Targeted random testing results

Inv.	Source	Description	Broken by rand testing? 10c       50c     100c
cr1	spec	W_YTD = sum(D_YTD)	✓	✓	✓
cr2	spec	D_NEXT_O_ID-1=max(O_ID)=max(NO_O_ID)	✕	✕	✓
cr3	spec	max(NO_O_ID)-min(NO_O_ID)+1=[number of corresponding rows in NEW-ORDER]	✕	✕	✓
cr4	spec	sum(O_OL_CNT) = [number of corresponding rows in ORDER-LINE]	✓	✓	✓
cr5a	spec	For any row in NEW_ORDER there is a row in ORDER with O_CARRIER_ID set to null	✕	✕	✕
cr5b	spec	For any row in ODER with O_CARRIER_ID set as null there is row in NEW_ORDER	✕	✕	✕
cr6	spec	O_OL_CNT is equal to the number of rows in ORDER-LINE for that order	✓	✓	✓
cr7a	spec	O_CARRIER_ID=null if all rows in ORDER_LINE have OL_DELIVERY_D=null	✕	✕	✕
cr7b	spec	all rows in ORDER_LINE have OL_DELIVERY_D=null if O_CARRIER_ID=null	✕	✕	✕
cr8	spec	W_YTD = sum(H_AMOUNT)	✓	✓	✓
cr9	spec	D_YTD = sum(H_AMOUNT)	✓	✓	✓
cr10	spec	C_BALANCE = sum(OL_AMOUNT) - sum(H_AMOUNT)	✕	✓	✓
cr11	spec	(count(*) from ORDER) - (count(*) from NEW-ORDER) = 2100	✓	✓	✓
cr12	spec	C_BALANCE + C_YTD_PAYMENT = sum(OL_AMOUNT)	✕	✓	✓
ncr1	anlys	s_ytd=sum_exec(ol_quantity)	✕	✓	✓
ncr2	anlys	sum(s_order_cnt) = sum_exec(o_ol_cnt)	✕	✓	✓
ncr3	anlys	In the absence of DELIVERY, (c_ytd_payment+c_balance) must be constant for each cust	✕	✕	✕
ncr4	anlys	c_payment_cnt = initial_value + #PAYMENT on that customer in execution log	✕	✕	✕
ncr5	anlys	sum(C_DELIVERY_CNT) = initial_value + number of times DELIVERY is called	✓	✓	✓
ncr6	anlys	ratio of different o_carrier_id must respect the corresponding ratio on DELIVERY’s inputs	✕	✓	✓
ncr7	anlys	H_AMOUNT for each customer = sum(payment_amounts) for that customer in log	✕	✕	✕

Table 4. Table 4 . Internal serializability anomalies found on TPC-C

#	Len	Txns	Tables	Type	Broken Invariants	Anlys	Rply
1	4	pm/pm	warehouse	lost update	cr1, cr8	5s	7s
2	4	pm/pm	district	lost update	cr1, cr9	19s	12s
3	4	no/no	district	lost update	cr2, …, cr7, cr11, ncr1	15s	13s
4	4	dv/dv	customer	lost update	cr10, cr12, ncr5	171s	12s
5	4	dv/dv	new_order	lost update	ncr6	16s	6s
6	4	pm/pm	history	lost update	cr8,cr9, cr10, ncr7	125s	11s
7	4	no/no/no	stock	unclassified	ncr1, ncr2	697s	13s
8	4	no/no	stock	lost update	ncr1, ncr2	70s	12s
9	4	dv/os/no	order_line	unclassified	none	11s	12s
10	4	dv/os/no	order_line	unclassified	none	9s	15s
11	4	dv/pm	customer	lost update	none	231s	9s
12	4	dv/pm	customer	lost update	ncr3,ncr4	12s	11s
13	4	pm/pm	customer	lost update	ncr3,ncr4,cr12	108s	8s

Equations

$$\displaystyle\footnotesize\begin{array}[]{lcl}\lx@intercol\hfil{a}\in\mathtt{Arguments}\qquad{v}\in\mathtt{Variables}\qquad{f}\in\mathtt{Fields}\qquad{A}\in\{\tt{min,max}\}\hfil\lx@intercol\\ \lx@intercol\hfil{T}\in\mathtt{TxnNames}\qquad\oplus\in\{+,-,\times,/\}\qquad\odot\in\{<,\leq,=,>,\geq\}\qquad\circ\in\{\wedge,\vee\}\hfil\lx@intercol\\ \\ \phi&\coloneqq&{e\odot e}~{}\mid~{}\neg\phi~{}\mid~{}\phi\circ\phi~{}\mid~{}\mathtt{true}~{}\mid~{}\mathtt{false}\\ {e}&\coloneqq&\mathbb{Z}~{}\mid~{}{a}~{}\mid~{}{e\oplus e}~{}\mid~{}{\mathtt{any(\phi)}}~{}\mid~{}{\mathtt{iter}}~{}\mid~{}{\mathtt{size}(v)}~{}\mid~{}{\mathtt{proj}(f,v,e)}~{}\mid~{}{\mathtt{this}.f}\\[0.85358pt] {q}&\coloneqq&{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}{f}\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{AS}\;}{v}\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}{\phi}~{}\mid~{}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}{A(f)}\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{AS}\;}{v}\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}{\phi}~{}\mid{}\\[-0.85358pt] &&{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SET}\;}{f=e}\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}{\phi}~{}\mid~{}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{INSERT}\;}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{VALUES}\;}{\overline{f}=\overline{e}}~{}\mid~{}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{DELETE}\;}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}{\phi}\\[0.56905pt] {c}&\coloneqq&{q}~{}\mid~{}\mathtt{if(\phi)}\{c\}~{}\mid~{}\mathtt{iterate}(e)\{c\}~{}\mid~{}c;c~{}\mid~{}\mathtt{skip}\\ \mathbb{P}&\coloneqq&{\overline{T(\overline{a})\{c\}}}\penalty 50\hskip 20.00003pt plus 0.5fil minus 5.0pt\end{array}$$

$$\footnotesize\frac{\begin{array}[]{c}T(a)\{c\}\in\mathbb{P}\qquad n\in\mathbb{Z}\par\end{array}}{\begin{array}[]{c}{\Sigma,\Gamma\xrightarrow{}\Sigma,\Gamma\!\cup\!\{c[a/n]\}}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}\Sigma,c\xrightarrow{}\Sigma^{\prime},c^{\prime}\par\end{array}}{\begin{array}[]{c}{\Sigma,\{c\}\!\cup\!\Gamma\xrightarrow{}\Sigma^{\prime},\{c^{\prime}\}\!\cup\!\Gamma}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}\\ \end{array}}{\begin{array}[]{c}{\Sigma,\mathtt{skip};c\xrightarrow{}\Sigma,c}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}{\Sigma,c\xrightarrow{}\Sigma^{\prime},c^{\prime\prime}\par}\end{array}}{\begin{array}[]{c}{\Sigma,c;c^{\prime}\xrightarrow{}\Sigma^{\prime},c^{\prime\prime};c^{\prime}}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}\Sigma\equiv(\mathtt{str},\mathtt{ar},\mathtt{vis})\quad p\in\mathcal{P}\\ \sigma=\Delta(\mathtt{ar},\mathtt{str}(p))\quad\sigma,\phi\Downarrow\mathsf{true}\par\end{array}}{\begin{array}[]{c}{\Sigma,\mathtt{if}~{}\phi~{}\{c\}\xrightarrow{}\Sigma,c}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}\Sigma\equiv(\mathtt{str},\mathtt{ar},\mathtt{vis})\quad p\in\mathcal{P}\\ \sigma=\Delta(\mathtt{ar},\mathtt{str}(p))\quad\sigma,\phi\Downarrow\mathsf{false}\par\end{array}}{\begin{array}[]{c}{\Sigma,\mathtt{if}~{}\phi~{}\{c\}\xrightarrow{}\Sigma,\mathtt{skip}}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}\Sigma\equiv(\mathtt{str},\mathtt{ar},\mathtt{vis})\quad p\in\mathcal{P}\\ \sigma=\Delta(\mathtt{ar},\mathtt{str}(p))\quad\sigma,e\Downarrow n\par\end{array}}{\begin{array}[]{c}{\Sigma,\mathtt{iterate}(e)\{c\}\xrightarrow{}\Sigma,\mathtt{concat}(n,c)}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}q\equiv{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}f\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{AS}\;}x\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}\phi\ \qquad{p}\in\mathcal{P}\qquad\sigma=\Delta(\mathtt{ar},\mathtt{str}({p}))\\ {\varepsilon_{1}}=\{{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd}(r,f)}\;|\;\sigma,\phi[\mathtt{this}.f^{\prime}\mapsto r(f^{\prime})]\Downarrow\texttt{true}\}\qquad{\varepsilon_{2}}={\{{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd}(r,f^{\prime})}\;|\;f^{\prime}\in\mathcal{F}(\phi)\}}\qquad\varepsilon=\varepsilon_{1}\cup\varepsilon_{2}\\ \mathtt{vis}^{\prime}=\mathtt{vis}~{}\cup~{}\{(\eta,\eta^{\prime})~{}\mid~{}\eta^{\prime}\in\varepsilon\wedge\eta\in\mathtt{str}({p})\}\qquad\;\;\mathtt{ar}^{\prime}=\mathtt{ar}~{}\cup~{}\{(\eta,\eta^{\prime})~{}\mid~{}\eta^{\prime}\in\varepsilon\wedge\eta\in\mathtt{str}\}\par\end{array}}{\begin{array}[]{c}{(\mathtt{str},\mathtt{vis},\mathtt{ar}),q\xrightarrow{}(\mathtt{str}[{p}{\mapsto}\mathtt{str}({p})\cup\varepsilon],\mathtt{vis}^{\prime},\mathtt{ar}^{\prime}),\texttt{skip}}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}q\equiv{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SET}\;}f=v\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}\phi\qquad{p}\in\mathcal{P}\qquad\sigma=\Delta(\mathtt{ar},\mathtt{str}({p}))\qquad\sigma,v\Downarrow n\\ {\varepsilon_{1}}=\{{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{wr}(r,f,n)}\;|\;\sigma,\phi[\mathtt{this}.f^{\prime}\mapsto r(f^{\prime})]\Downarrow\texttt{true}\}\qquad{\varepsilon_{2}}=\{{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd}(r,f^{\prime})}\;|\;f^{\prime}\in\mathcal{F}(\phi)\}\qquad\varepsilon=\varepsilon_{1}\cup\varepsilon_{2}\\ \mathtt{vis}^{\prime}=\mathtt{vis}~{}\cup~{}\{(\eta,\eta^{\prime})~{}\mid~{}\eta^{\prime}\in\varepsilon\wedge\eta\in\mathtt{str}({p})\}\qquad\mathtt{ar}^{\prime}=\mathtt{ar}~{}\cup~{}\{(\eta,\eta^{\prime})~{}\mid~{}\eta^{\prime}\in\varepsilon\wedge\eta\in\mathtt{str}\}\end{array}}{\begin{array}[]{c}{(\mathtt{str},\mathtt{vis},\mathtt{ar}),q\xrightarrow{}(\mathtt{str}[{p}{\mapsto}\mathtt{str}({p})\!~{}\cup~{}\!\varepsilon],\mathtt{vis}^{\prime},\mathtt{ar}^{\prime}),\texttt{skip}}\end{array}}$$

$$\footnotesize\frac{\begin{array}[]{c}q\equiv{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{DELETE}\;}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}\phi\qquad{p}\in\mathcal{P}\qquad\sigma=\Delta(\mathtt{ar},\mathtt{str}({p}))\\ {\varepsilon_{1}}=\{{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{wr}(r,{alive},0)}\;|\;\sigma,\phi[\mathtt{this}.f^{\prime}\mapsto r(f^{\prime})]\Downarrow\texttt{true}\}\qquad\varepsilon_{2}=\{{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd}(r,f^{\prime})}\;|\;f^{\prime}\in\mathcal{F}(\phi)\}\qquad\varepsilon=\varepsilon_{1}\cup\varepsilon_{2}\\ \mathtt{vis}^{\prime}=\mathtt{vis}~{}\cup~{}\{(\eta,\eta^{\prime})~{}\mid~{}\eta^{\prime}\in\varepsilon\wedge\eta\in\mathtt{str}({p})\}\qquad\mathtt{ar}^{\prime}=\mathtt{ar}~{}\cup~{}\{(\eta,\eta^{\prime})~{}\mid~{}\eta^{\prime}\in\varepsilon\wedge\eta\in\mathtt{str}\}\end{array}}{\begin{array}[]{c}{(\mathtt{str},\mathtt{vis},\mathtt{ar}),q\xrightarrow{}(\mathtt{str}[{p}{\mapsto}\mathtt{str}({p})\!~{}\cup~{}\!\varepsilon],\mathtt{vis}^{\prime},\mathtt{ar}^{\prime}),\texttt{skip}}\end{array}}$$

$${\phi }_{\mathbb{C}}\stackrel{\Delta }{\equiv }{\phi }_{\text{context}}\wedge {\phi }_{\text{db}}\wedge {\phi }_{\text{dep→}}\wedge {\phi }_{\text{→dep}}\wedge \underset{\begin{array}{c}0\le i\le ma{x}_p\\ 2\le j\le ma{x}_t\\ 3\le k\le ma{x}_c\end{array}}{\bigvee }{\phi }_{\text{anomaly}}^{i,j,k}$$

$${\phi }_{\text{anomaly}}^{i,j,k}\stackrel{\Delta }{\equiv }\exists t_1,...,t_i.\underset{1\le m\le i}{\bigwedge }{\phi }_{\text{ser}}(t_m)\wedge \exists t_1^{\prime },...,t_j^{\prime }.{\phi }_{\text{cycle}}^k(t_1^{\prime },...,t_j^{\prime })\wedge \underset{\begin{array}{c}1\le m\le i\\ 1\le n\le j\end{array}}{\bigwedge }{\phi }_{\text{order}}(t_m,t_n^{\prime })$$

$${\phi }_{\text{cycle}}^k(t_1,...,t_j)\stackrel{\Delta }{\equiv }$$

$$\mathtt{D}(q_1,q_2)\wedge \mathtt{D}(q_{k-1},q_k)\wedge \mathtt{ST}(q_1,q_k)\wedge \underset{2\le m<k-1}{\bigwedge }(\mathtt{ST}(q_m,q_{m+1})\vee \mathtt{D}(q_m,q_{m+1}))$$

$$\underset{q_1,q_2\in \mathfrak{q}}{\bigwedge }\underset{r\in \mathfrak{r}}{\bigwedge }\neg \mathtt{WR}(q_2,q_1)\Rightarrow {\mathtt{val}}^f(q_1,r)={\mathtt{init}}^f(r)$$

$$\forall q_1,q_2.\mathtt{ar}(q_1,q_2)\wedge (\tau (q_1)=\tau (q_2))\Rightarrow \mathtt{vis}(q_1,q_2)$$

$$\footnotesize\frac{\begin{array}[]{c}{q\equiv{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}f\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{AS}\;}x\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}\phi}\\ {q^{\prime}\equiv{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SET}\;}f=v\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}\phi^{\prime}}\\ {\mathtt{txn}(q)=t\qquad\mathtt{txn}(q^{\prime})=t^{\prime}\qquad t\not\eq t^{\prime}}\end{array}}{\begin{subarray}{c}\mu^{\mathtt{RW}\rightarrow}_{q,q^{\prime}}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\exists r.\;{[\![\phi]\!]^{\mathbb{B}}_{t,r}}\wedge{[\![\phi^{\prime}]\!]^{\mathbb{B}}_{t^{\prime},r}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\wedge\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathtt{Alive}{}(r,q)\wedge}\\ {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathtt{Alive}{}(r,q^{\prime})\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\wedge\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{[\![\Lambda(q)]\!]^{\mathbb{B}}_{t}}\wedge{[\![\Lambda(q^{\prime})]\!]^{\mathbb{B}}_{t^{\prime}}}}\end{subarray}}$$

$${\phi }_{\text{dep→}}\stackrel{\Delta }{\equiv }\underset{\mathtt{D}\in \{\mathtt{WR},\mathtt{RW},\mathtt{WW}\}}{\bigwedge }\underset{q,q^{\prime }\in \mathfrak{q}}{\bigwedge }(\mathtt{D}(q,q^{\prime })\Rightarrow {\mu }_{q,q^{\prime }}^{\mathtt{D}\to })$$

$$\footnotesize\frac{\begin{array}[]{c}{q\equiv{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}f\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{AS}\;}x\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}\phi}\\ {q^{\prime}\equiv{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SET}\;}f=v\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}\phi^{\prime}}\\ {\mathtt{txn}(q)=t\qquad\mathtt{txn}(q^{\prime})=t^{\prime}}\qquad t\not\eq t^{\prime}\end{array}}{\begin{subarray}{c}\mu^{\rightarrow\mathtt{WR}}_{q^{\prime},q}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\mathtt{vis}(q^{\prime},q)\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\wedge\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\exists r.\;{[\![\phi]\!]^{\mathbb{B}}_{t,r}}\wedge{[\![\phi^{\prime}]\!]^{\mathbb{B}}_{t^{\prime},r}}\wedge}\\ {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\qquad\mathtt{Alive}{}(r,q)\wedge\mathtt{Alive}{}(r,q^{\prime})\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\wedge\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}{[\![\Lambda(q)]\!]^{\mathbb{B}}_{t}}\wedge{[\![\Lambda(q^{\prime})]\!]^{\mathbb{B}}_{t^{\prime}}}}\end{subarray}}$$

$${\phi }_{\text{→dep}}\stackrel{\Delta }{\equiv }\underset{\mathtt{D}\in \{\mathtt{WR},\mathtt{RW},\mathtt{WW}\}}{\bigwedge }\underset{q_1,q_2\in \mathfrak{q}}{\bigwedge }({\mu }_{q_1,q_2}^{\to \mathtt{D}}\Rightarrow \mathtt{D}(q_1,q_2))$$

Full text

To address these challenges, this paper presents a novel testing framework for detecting serializability violations in (SQL) database-backed Java applications executing on weakly-consistent storage systems. We manifest our approach in a tool named clotho, that combines a static analyzer and a model checker to generate abstract executions, discover serializability violations in these executions, and translate them back into concrete test inputs suitable for deployment in a test environment. To the best of our knowledge, clotho  is the first automated test generation facility for identifying serializability anomalies of Java applications intended to operate in geo-replicated distributed environments. An experimental evaluation on a set of industry-standard benchmarks demonstrates the utility of our approach.

Static Analysis, Serializability, Weak Consistency

††copyright: none††ccs: Software and its engineering Formal software verification

1. Introduction

Realistic SQL-based databases typically have sophisticated structural relationships (schemas), and clients must ensure that every possible use of a database operation preserves these relationships. Testing that client applications do so is particularly challenging, as the operations they perform depends on the control structure and initial state of the application. Generating appropriate test inputs to discover violations of these relationships necessarily requires reasoning about the behaviors of these operations in the context of the program’s execution. Further complicating matters is that the initial state of the database itself needs to be chosen so that tests expose interesting integrity and assertion violations.

One important simplification that helps reduce the complexity of developing a useful testing strategy for database applications is to treat database transactions as serializable, restricting the set of interleavings that must be considered to those that maintain transaction atomicity and isolation.

As an example, consider the transactions txnWrite and txnRead from the simple pseudocode application presented in Figure 1. Transaction txnWrite updates rows from table1 and table2 with the value val where both rows are accessed using the given id. Similarly, transaction txnRead reads rows from both tables that are associated with the given id and asserts that they are equal. This is a very common pattern in database applications, e.g. when backup versions of tables are maintained.

Under the guarantees afforded by serializability, an automated testing framework need only consider interleavings of entire transactions in order to completely cover the state of possible valid concurrent executions. This is because interleavings that expose intermediate transaction states (failure of atomicity), or which allow an already executing transaction to observe the effects of other completed transactions (failure of isolation) are prohibited, and hence the space of executions that must be considered to validate an application’s correctness is greatly reduced. While conceptually simple to reason about, maintaining serializability imposes strong constraints on the underlying database and storage system that hinders performance at scale. In large distributed environments, especially those use replication to improve latency, the cost of enforcing serializability can be prohibitive, requiring global synchronization to enforce atomic updates and maintain a single global view of database state. Consequently, modern-day cloud systems often implement and support storage abstractions that provide significantly weaker consistency and isolation guarantees than those necessary to enforce serializable executions of database programs.

For example, when the transactions in the above example are executed on a replicated database systems offering eventual consistency (EC), each individual read and write operation can be executed independently making the space of possible executions significantly larger than before. Not surprisingly, new anomalous executions are also introduced under such weak semantics. These executions can trigger assertion violations that would otherwise not happen, for example, if both read operations occur between the two writes and vice versa. To make matters worse, these violations are only triggered if the initial state of the database for both rows differs from what the txnWrite transaction is writing. Given these complexities, the obvious question that arises is whether a database application, originally written assuming the underlying storage system provides strong consistency and isolation guarantees, will still operate correctly (i.e., behave as though serializability were still maintained) in weaker environments where these guarantees are no longer assured.

Given these concerns, the need for a practical automated testing framework for database applications executing in weakly consistent environments becomes is particularly acute. This paper takes an important first step towards addressing this need by presenting a tool (clotho) that systematically and efficiently explores abstract executions of database-backed Java programs for serializability violations, assuming a weakly-consistent storage abstraction. To do so, we employ a bounded model-checking serializability violation detector that is parameterized by a specification of the underlying storage model. Abstract executions capture various visibility and ordering relations among read and write operations on the database generated by queries; the structure of these relations is informed by the underlying data consistency model. Potential serializability violations in an abstract execution manifest as cycles in a dependency graph induced by these relations. When such violations are discovered, clotho synthesizes concrete tests that can be used to drive executions that manifest the problem for assessment by application developers.

Through our experimental evaluation, we demonstrate that clotho  can efficiently, automatically, and reliably detect and concretely manifest serializability anomalies of database programs executing on top of weakly-consistent data stores. An important by-product of its design is that clotho  does not bake-in any specific consistency or isolation assumptions about the underlying storage infrastructure, allowing users to strengthen visibility and ordering constraints as desired.

The contributions of this paper are as follows:

1. (1)

We present an abstract representation and a novel encoding of database applications capable of modeling a diverse range of weak guarantees offered by real-world distributed databases. 2. (2)

We develop a search algorithm capable of effectively identifying unique serializability anomalies accompanied by a complete execution history that leads to that anomaly. 3. (3)

We bridge the gap between abstract and concrete executions by developing a front-end compiler and a test administration environment for database-backed Java applications that admits a wide range distributed database system behaviors and features. 4. (4)

We demonstrate the utility of our approach on a set of standard benchmarks and present a comprehensive empirical study comparing clotho’s effectiveness against a tuned dynamic (random) testing approach.

The remaining of this paper is organized as follows. A brief overview of our solution using a motivational example is presented in Section 2. Sections 3 and 4 respectively define the operational semantics of our abstract programming model and the notion anomalies in that model. Section 5 presents our precise encoding of anomalies in database applications. Section 6 introduces our effective search algorithm and the implementation details of clotho. We present two sets of empirical results in Section 7 showing applicability and effectiveness of our approach and its performance compared to a state of the art dynamic testing framework. Lastly we summarize related works in section 8 before concluding the paper in Section 9.

2. Overview

TPC-C is a canonical Online Transaction Processing (OLTP) benchmark application that emulates a warehouse management and order entry/delivery system. The benchmark was originally designed to validate a mixture of requirements (including transactional safety) expected from relational database systems (tpc, 2010). To illustrate our approach, consider Figure 2 (left) that presents a pruned code snippet implementing a transaction from TPC-C named payment. The snippet shows a procedure that updates the total number of successful payments from a customer, where the customer’s associated row from $\mathsf{CUST}$ table is initially retrieved (lines 3-5) and is then rewritten with an incremented value (lines 10-13). Now, consider the deployment of TPC-C on a replicated cloud database presented in Figure 2 (right) where two payment transactions are concurrently executed on weakly-consistent replicas A and B that do not enforce serializability. The initial database state in this execution consists of the table $\mathsf{CUST(id,c\_pay\_cnt)}$ with a single row $\mathsf{r_{0}\!:=\!(10,50)}$. Additionally, the transaction instances txn1 and txn2 are given arguments such that both transactions internally access and update $\mathsf{r_{0}}$. Both transactions concurrently read the initial value of $\mathsf{r_{0}}$ at different replicas (depicted as incoming labeled arrows to the transaction) and then both locally replace it with updated values $\mathsf{r_{0}^{A}\!:=\!(10,51)}$ and $\mathsf{r_{0}^{B}\!:=\!(10,51)}$ (depicted by outgoing labeled arrows). Each replica then propagates the locally updated row to the other replica (depicted by dashed arrows), where the written value at B supersedes value written at A, leaving the database with the final state of $\mathsf{r_{0}\!:=\!(10,51)}$.

Because the above scenario is not equivalent to any sequential execution of the two transaction instances, this behavior is properly classified as a serializability anomaly. Unfortunately, buggy executions like this are often permitted by modern cloud databases which drop support for atomic and isolated transactions in favor of fault tolerance, scalability, and availability. This foists a significant challenge onto OLTP application developers, requiring them to detect the potential anomalies that can occur in a particular application and validate them against different database systems in order to diagnose and remedy undesired behaviors.

Developing a testing framework that discovers anomalies of this kind is challenging, however. Part of the problem is due to the large set of possible interleaved executions that are possible. For example, the serializability anomaly described above is crucially dependent on the exact order in which operations are transmitted to different replicas and occurs only when the same customer (out of 30000 customers) is accessed by both instances. Combined with the possibility that operations within a single transaction instance can also be unreachable in some program paths, and that the execution paths taken are highly dependent on the initial database state, the chance of randomly encountering conflicts of this kind becomes even smaller.

In addition to the above challenges, the high computational cost of detecting serializability anomalies at runtime(Papadimitriou, 1979) means that practical testing methods often leverage a set of user-provided application-level invariants and assertions to check for serializability failure-induced violations. But, these assertions are often underspecified. For example, the anomaly depicted in Figure 2, which most developers would classify as a bug, does not directly violate any of the twelve officially specified invariants of the TPC-C benchmark.

clotho

To overcome these challenges, we have developed a principled test construction and administration framework called clotho that targets database applications intended to be deployed on weakly-consistent storage environments. Our approach reasons over abstract executions of a database application. Serializability anomalies manifest as cycles in such executions. clotho translates the discovered abstract anomalies back to concrete executions which can be then automatically replayed and validated.

The test configuration details produced by clotho include

(i) concrete transaction instances and parameters,

(ii) concurrent execution schedules,

(iii) network partitionings throughout the execution,

(iv) session management, and

(v) initial database state.

Our test administration framework allows arbitrary distributed or centralized databases to be plugged into its managed containers. Given a set of test configurations, clotho  automatically executes multiple application instances according to each test configuration, in order to effectively manifest and validate the intended serializability anomaly against a variety of actual database systems.

Figure 3 presents an overview of clotho’s design. clotho’s static analysis backend is based on an intermediate abstract representation of database programs, automatically generated from input Java source code via a front-end compiler. The abstract program representation is passed to an encoding engine (➊) that constructs FOL formulae that capture the necessary conditions under which a dependency cycle forms over instances of database operations. This fine-grained SAT representation of the problem is then passed to an off-the-shelf theorem prover (➋); responses with a satisfying solution (➌) are recorded and re-encoded into the context to be passed to the solver again with the intention of finding another anomaly. This iterative approach allows for a complete search of the space of anomalies within a configurable cycle-detection length. Once the bounded space is completely covered (i.e., there are no more satisfying solutions), test configuration files (➎) are generated from the collected abstract anomalies that provide details about concrete executions that can potentially manifest the intended anomaly. Additionally, the source code of the program is passed through a code annotator (➍,➏) which returns multiple instances of the source code including concrete function parameters and annotations on all database access operations.

For example, A1.conf in Figure 4 is the configuration file that specifies the initial state of the database needed to manifest the previously discussed anomaly on the $\mathsf{CUST}$ table. It also specifies the execution order of operations from the two payment transaction instances and the network status at each logical step (T1 to T4). The configuration is completed with annotated source code that specifies the input parameters to the payment transaction and marks the database operations with their relative local orders (A​1_Ins1.java and A​1_Ins2.java).

clotho  additionally offers a fully automated test administration framework, which consists of centrally managed wrappers for arbitrary database drivers which can temporarily block database access requests in order to enforce a specific execution order (➐). Similarly, database replicas are deployed in managed containers, communication among which can be throttled (➑) in order to induce temporary network partitionings. This enables clotho  to efficiently administer a wide range of statically constructed potential serializability anomalies, allowing developers to witness and study undesired application behaviors that frequently occur in the wild but which are extremely difficult to catch using existing testing frameworks.

3. Abstract Model

$\mathcal{AR}$ is the target language of clotho’s front-end compiler and is designed to capture key features of database-backed applications, including schemas, transactions, and data retrieval and modification operations. We begin by introducing the syntax of $\mathcal{AR}$ programs and presenting an overview of its semantics. A complete account of the semantics of $\mathcal{AR}$ can be found in (Rahmani et al., 2019).

3.1. Abstract Representation

$\mathcal{AR}$ programs are parameterized over a fixed schema that describes a set of database tables. Each table is defined as an ordered list of field names from the set $\mathtt{Fields}$. In order to uniquely identify individual records, each table is assigned a non-empty subset of its fields that acts as a primary key. For simplicity, our presentation assumes a single table with only integer-valued fields, but clotho relaxes this restriction and supports an arbitrary number of tables and other Java primitive types.

The syntax of $\mathcal{AR}$ is given in Figure 5. Programs in $\mathcal{AR}$ consist of a set of parameterized transactions, each of which has a unique name drawn from the set $\mathtt{TxnNames}$. The body of a transaction is a command, denoted by $c$, which can be either a database query, a guarded command, a loop, or a sequence of commands. Database queries, denoted by $q$, retrieve ({\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}\!\!) or modify ({\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}\!\!, {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{INSERT}\;}\!and {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{DELETE}\;}\!\!\!\;) records in the database. The result of each {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}\!query is stored as an ordered list of records in a unique variable $v$.

Boolean expressions of $\mathcal{AR}$, denoted by $\phi$, consist of standard arithmetic comparison and boolean operators.

Arithmetic expressions, denoted by $e$, include integer constants, transaction arguments, arithmetic operations, non-deterministic values, iteration counters, the number of results held in a variable, and field accesses, denoted as proj. For example, $\mathtt{proj}(age,v,3)$ returns the value of $age$ field in the third record stored in the variable $v$. The expression $\mathtt{any}(\phi)$ can evaluate to any value satisfying the predicate ${\phi}$ and is used to model source language features such as unknown function calls and updates to program variables within non-deterministic loops. The iteration counter $\mathtt{iter}$ represents the number of times a loop has been repeated during a program execution. Field names can also appear in an arithmetic expression as $\mathtt{this}.f$, which can only be used in conditionals in the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}clauses of queries and denote the field values of records that are being evaluated in that clause.

As an example, Figure 6 presents a Java code snippet and its $\mathcal{AR}$ encoding. The program scans a table and retrieves all records representing employees younger than 35 and then increases their salaries within a while loop.

3.2. System Configurations

The operational semantics of $\mathcal{AR}$ is defined by a small-step reduction relation, $\rightarrow~{}\subseteq~{}\Sigma\times\Gamma\times\Sigma\times\Gamma$, on the current system state of the underlying database, denoted as $\Sigma$, and a set of concurrently running transaction instances, denoted as $\Gamma$. Mimicking the execution of database-backed programs on replicated stores, $\mathcal{AR}$ programs are interpreted on a finite number of partitions, each of which has its own copy of the database. System states are therefore represented as a triple, $(\mathtt{str},\mathtt{ar},\mathtt{vis})$, which holds a history of database reads and writes executed at each partition and a pair of orderings on these events. Executing a database operation generates a set of read effects, {\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd}(r,f)}, witnessing that the field $f$ of record $r$ was accessed, and write effects, {\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{wr}(r,f,n)}, recording that the field $f$ of record $r$ was set to the value $n$. Read effects may optionally include the value read, e.g. {\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd}(r,f,n)}. Each effect is implicitly tagged with a unique identifier, so that we can distinguish, for example, between the same field being read by multiple queries. Given a finite set $\mathcal{P}$ of partition names, $\mathtt{str}$ represents the set of effects generated on each partition as a mapping from partition names to disjoint sets of effects. When used without an argument, $\mathtt{str}$ refers to the complete set of effects from all partitions, i.e. $\mathtt{str}:=\underset{\scriptscriptstyle{p}\in\mathcal{P}}{\bigcup}\mathtt{str}(p)$.

The $\mathtt{ar}$ component of the system state records the exact sequence of database operations that have been executed as an order between effects called an arbitration relation. This relation is however too coarse-grained to capture causal relationships between effects: an effect $\eta_{1}$ created at partition $p_{1}$, does not necessarily influence the execution of a query at partition $p_{2}$ that creates another effect $\eta_{2}$, even though $\eta_{1}$ may be arbitrated before $\eta_{2}$. The system state therefore maintains a more refined visibility relation, $\mathtt{vis}\subseteq\mathtt{ar}$, between effects, which only relates two effects if one actually witnesses the other at the time of creation.

It is possible to construct a current copy, or local view, of the database at each partition $p$ from a system state by “applying” the effects of query operations stored in $\mathtt{str}(p)$ according to the order in $\mathtt{ar}$. Such local views are denoted as $\sigma$ and are modeled as functions from primary keys to records. To ensure $\sigma$ is total, all tables include a special field, $alive\in\mathtt{Fields}$, whose value determines if a record is present in the table or not.

3.3. Operational Semantics

Figure 7 presents the rules defining the operational semantics of $\mathcal{AR}$ programs and commands. These rules are parameterized over an $\mathcal{AR}$ program, $\mathbb{P}$. Each step either adds a new instance to the set of currently running transactions via e-spawn111For simplicity of presentation, we assume the spawned transaction has only one argument or executes the body of one of the currently running transactions via e-step. The reduction rules for commands non-deterministically select a partition $p$ to execute the command on. This partition is used by an auxiliary function $\Delta$ to construct a local view of the database. These local views are used by evaluation relation for boolean and arithmetic expressions, $\Downarrow$. The rules for non-query commands are straightforward outside for the (e-iter) rule, which uses the $\mathtt{concat}(n,c)$ function that sequences $n$ copies of the command $c$, with any occurrences of iter being instantiated as expected.

The operational semantics for queries, presented in Figure 8, are more interesting. The key component of each of these rules is how they construct the set of new effects generated by each database operation, denoted by $\varepsilon_{1}$ and $\varepsilon_{2}$. In the rule for {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}\!\!, $\varepsilon_{1}$ includes appropriate read effects for each record satisfying the operation’s {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}condition. Any occurrence of $\mathtt{this}.f$ construct in the conditional $\phi$, is substituted with the value of the corresponding field in each record instance under evaluation. The rule for {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}similarly includes a write effect in $\varepsilon_{1}$ for each record satisfying the operation’s {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}condition. To model the deletion of records, the rule for {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{DELETE}\;}creates write effects with a constant value 0 for the $alive$ fields of all records that satisfy its {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}condition.

In order to capture the data accesses that occur during database-wide scans, all three rules include new read effects in $\varepsilon_{2}$ for every field that is predicated in the operation’s {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}clause. To do so, these rules use the auxiliary function $\mathcal{F}{}$ which extracts any such fields from boolean expressions. $\mathcal{F}{}$ additionally always includes the $alive$ field. As an example, the result of this function for the where clause from the query in Figure 6 is, $\mathcal{F}{(age<35)}=\{age,alive\}$.

Finally, each rule updates the $\mathtt{ar}$ (resp. $\mathtt{vis}$) relation to reflect the relationship between the newly created effects in $\varepsilon_{1}\cup\varepsilon_{2}$ and already existing effects in $\mathtt{str}$ (resp. $\mathtt{str}(p)$).

Figure 9 depicts a concrete example of two consecutive execution steps of the $\mathcal{AR}$ program from Figure 6. We depict an execution in which arithmetic expressions in the update operation ($q_{2}$) have been fully evaluated and both operations execute on the same partition. The example furthermore simplifies $\mathtt{str}$ to be just the set of effects in that particular partition. The initial system state encodes a database with three employee records, only two of which are alive. Because the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}query does not modify any database record, the local views $\sigma$ and $\sigma^{\prime}$ are identical.

The {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}query constructs a set of new effects in $\varepsilon_{2}$, including three read effects on the $age$ field of all records (since $age$ is constrained by its {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}clause) and three read effects on all $alive$ fields. Since only one of the witnessed records is both alive and satisfies the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}condition (i.e. $age<35$), a single read effect on that record, {\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd}(1,sal)}, is also created and included in $\varepsilon_{1}$. Similarly, $\varepsilon_{1}^{\prime}$ is defined for the execution of {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}query, which includes only a single write effect, {\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{wr}(1,sal,86)}, capturing the modified salary value for the previously selected employee record. $\varepsilon_{2}^{\prime}$ is empty, because the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}query uses the primary key of the table to access the record, without any database-wide scans. Note that both steps update the arbitration and visibility relations identically, since the example assumes only a single partition.

4. Serializability Anomalies

We now turn to the question of identifying and statically detecting undesirable executions in $\mathcal{AR}$ programs.

4.1. Serial(izable) Executions

An execution history of an $\mathcal{AR}$ program is simply a finite sequence of system configurations allowed by the reduction relation: $H\equiv(\Sigma_{0},\Gamma_{0})\rightarrow(\Sigma_{1},\Gamma_{1})\rightarrow\dots\rightarrow(\Sigma_{k},\Gamma_{k})$. An execution history is said to be valid, if all the spawned transactions are fully executed, i.e. $\Gamma_{0}$ and $\Gamma_{k}$ are both empty. A valid history furthermore requires $\Sigma_{0}$ to be consistent with any user-defined constraints on the initial state of a database, e.g. that certain tables are initially empty.

Following the literature on weak consistency and isolation semantics (Burckhardt et al., 2014) we can specify constraints on $\mathtt{vis}$ and $\mathtt{ar}$ relations which identify when system states are consistent with a particular database’s consistency guarantees. Table 1 presents some well-known instances of such guarantees and their specifications, where the predicate $\mathtt{st}$ simply relates effects created by queries from the same transaction instance. System states can be thought of as finite models for these guarantees. For example, a system state $\Sigma$ is said to be strictly serial if and only if $\Psi_{\textsc{ser}}$ is satisfied by $\Sigma$, denoted as $\Sigma\models\Psi_{\textsc{ser}}$. Since both $\mathtt{vis}$ and $\mathtt{ar}$ only grow during the execution of a program, if the final system state satisfies a particular specification, so will every system state in the execution history. We correspondingly extend this notion to execution histories and define a history $H$ to conform to a guarantee $\Psi$, denoted by $H\models\Psi$, if and only if $\Psi$ is satisfied in the final system state of $H$.

Although strictly serial executions are easily comprehensible and desirable, forcing the database to execute transactions one-by-one is unnecessarily restrictive and must usually rely on poorly scalable pessimistic locking mechanisms to enforce (Sivaramakrishnan et al., 2015). We instead define an execution history $H$ to be serializable if there exists another strictly serial history $H^{\prime}$, that is constructed by reordering execution steps of $H$, such that the final set of effects in both histories are equivalent.

For example, Figure 10 depicts the system states from a history $H$ (left), and another serial history $H^{\prime}$ (right) which is constructed by reordering steps ② and ③ of $H$. Although $H$ is not strictly serial (since both $\Psi_{\textsc{rc}}$ and $\Psi_{\textsc{rr}}$ are violated in $\Sigma_{4}$), its final set of events, $\mathtt{str}_{4}$, is the same as $\mathtt{str}_{4}^{\prime}$, the final set of events in $H^{\prime}$, modulo renaming of their implicit unique identifiers, and is therefore serializable.

Given an $\mathcal{AR}$ program, our goal is to statically find non-serializable execution histories that are consistent with the guarantees of the underlying database, i.e. to detect any serializability anomalies in the program. As a first step, observe that execution histories that contain a serializability anomaly can be decomposed into a serial execution history followed by a (usually smaller) non-serializable history. For instance, a non-serializable execution of ten transactions, may be decomposed into eight serially executed transactions which insert appropriate records into initially empty tables, needed for the remaining two (concurrently executed) transactions that actually manifest the serializability anomaly. Following this observation, clotho systematically prunes the space of all possible interleavings of transactions involved in lengthy anomalous executions and instead searches for independent serializability anomalies, defined as two disjoint histories $H_{1}\rightarrow H_{2}$, where $H_{1}$ is a valid serial execution and $H_{2}$ is a non-serializable execution history.

4.2. Internal and External Serializability

During the course of our experiments, we observed that a subset of serializability anomalies could be considered benign, in that they always leave the final state of the database indistinguishable from the state after some serializable execution of the same instances. As an example, consider the executions $H$ and $H^{\prime}$ presented in the left and right of Figure 11. $H$ consists of two instances of the $\mathsf{upd}$ transaction, which reads the variable $\mathsf{x}$ and then updates it to a fresh value. Similarly, $H^{\prime}$ includes two instances of the $\mathsf{inc}$ transaction which reads the value of $\mathsf{x}$ and then increments it by 10. Both of these executions are non-serializable as they manifest the classic lost update anomaly (Adya, 1999). However, only $H^{\prime}$ leaves the database in a state that is inconsistent with any serializable execution. We dub the sort of benign anomalies seen in $H$ external serializability anomalies.

The key observation is that it is possible for some of the read effects in a non-serializable execution to not impact a transaction’s control flow or any of later write effects. As a result, those read events could be excluded from the correctness analyses, in order to focus on internal serializability anomalies. Such anomalies are more harmful in that they leave the database state permanently diverged from a serializable execution state and should be carefully studied and addressed by developers. We denote such unused read effects as {\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd^{+}}}, which can be detected through a straightforward analysis of the source program. For these reasons, by default, clotho only identifies internal serializabilty anomalies.

4.3. Dependency Cycles

Following the approach of Adya et al. (2000), we reduce the problem of determining serializability of an execution history to the detection of cycles in the dependency graph of its final state. To this end, we first define three dependency relations over the set of effects in an execution state:

(i) Read dependency, $\mathtt{WR}$, which relates two effects if one witnesses a value in a record’s field that is written by the other.

(ii) Write dependency, $\mathtt{WW}$, which relates two effects if one overwrites the value written by the other.

(iii) Read anti-dependency, $\mathtt{RW}$, which relates two effects if one witnesses a value in a field that is later overwritten by the other effect.

For example, in Figure 11 (left) both dependency relations \mathtt{WW}({\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{wr}(x,f,1)},{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{wr}(x,f,2)}) and \mathtt{RW}({\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{rd^{+}}(x,f,0)},{\color[rgb]{0.19921875,0,0.3984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.19921875,0,0.3984375}\mathsf{wr}(x,f,2)}) hold.

Recalling that serializability is defined by a reordering of database operations, we lift the definition of the above dependency relations (and previously defined $\mathtt{vis}$ and $\mathtt{ar}$) from effects to query instances. Given an execution history $H$ and for all relations $\mathtt{R}\in\{\mathtt{ar},\mathtt{vis},\mathtt{RW},\mathtt{WR},\mathtt{WW}\}$, if effects $\eta$ and $\eta^{\prime}$ are created respectively by queries $q$ and $q^{\prime}$ in $H$, then $\mathtt{R}(\eta,\eta^{\prime})\Rightarrow\mathtt{R}(q,q^{\prime})$. This lifting is necessary because in order to determine serializability of a history $H$, the execution order of queries in $H$ must be altered such that the new history $H^{\prime}$ is serial (see section 4.1). Intuitively, such reordering cannot not exist when some queries in $H$ are cyclically dependent to each other.

We are now ready to formally define the dependency graph $G$ of a system state $\Sigma$ as a directed multigraph whose nodes are query instances (that created the effects in $\Sigma$) and edges are in $\{\mathtt{RW},\mathtt{WR},\mathtt{WW},\mathtt{ST}\}$. The relation $\mathtt{ST}$ is simply defined to relate query instances originated from the same transaction. We further define a valid dependency cycle as a 2-regular subgraph of $G$, that includes at least two dependency edges (i.e. $\mathtt{WR},\mathtt{RW},\mathtt{WW}$) which are connected by an $\mathtt{ST}$ edge. We do not allow dependency edges between query instances from the same transaction and w.l.o.g assume they are replaced with $\mathtt{ST}$ edges. Figure 12 presents schematic examples of valid and invalid cycle structures in dependency graphs.

We conclude this section by describing the connection between the dependency graph of the final state in an execution to its serializability. To this end, an execution history $(\Sigma_{0},\Gamma_{0})\rightarrow\dots\rightarrow(\Sigma_{k},\Gamma_{k})$ is serializable if and only if $\Sigma_{k}$’s dependency graph is acyclic (Adya et al., 2000). As we will explain in the next section, this connection enables clotho to encode potential dependency cycles in a given $\mathcal{AR}$ program into a decidable first-order formula and statically construct an execution history that manifests a serializability anomaly.

5. Static Anomaly Generation

We now turn to the question of how to statically detect dependency cycles and construct independent serializability anomalies in $\mathcal{AR}$ programs. We adopt the approach presented in (Nagar and Jagannathan, 2018) and (Brutschy et al., 2018) and reduce (the bounded version of) our problem to checking the satisfiability of an FOL formula $\varphi_{\mathbb{C}}$, constructed from a given $\mathcal{AR}$ program. This formula includes variables for each of the dependency, visibility, and arbitration constraints that can appear during the program’s execution, and is designed such that the assignments to these variables in any satisfying model can be used to reconstruct an anomalous execution of the original program. This allows us to use an off-the-shelf SMT solver to efficiently check for anomalies in the given program. Given bounds $max_{p}$, $max_{t}$ and $max_{c}$, the shape of the full formula is a conjunction of five clauses, each encoding a different aspect of the program:

[TABLE]

In the above formula, $\varphi_{\text{{context}}}$ represents a set of constraints on variables and functions, which ensures that a satisfying assignment corresponds to a valid execution of any database program. Clause $\varphi_{\text{{DB}}}$ enforces a set of user-defined validity constraints on database records. Clauses \varphi_{\text{{dep\rightarrow}}} and \varphi_{\text{{\rightarrowdep}}} capture the necessary and sufficient conditions for establishing dependency relations between queries in the given program.

Following our discussion of execution histories which manifest an independent serializability anomaly in section 4.1, $\varphi_{\text{{anomaly}}}^{i,j,k}$ forces the solver to instantiate a serial execution of $i$ transactions that lead to concurrent execution of $j$ transactions within which a dependency cycle of length $k$ is formed. Figure 13 presents an example of such an instantiation where $i=3$, $j=2$ and $k=4$. The remainder of this section lays the formal groundwork for the encoding and explains each of the above clauses in more detail. The complete details of the encoding can be found in the extended version of this paper (Rahmani et al., 2019).

5.1. Components of FOL Encoding

We assume uninterpreted sorts for transactions ($\mathfrak{t}$), queries ($\mathfrak{q}$), partitions ($\mathfrak{p}$), and records ($\mathfrak{r}$). We define functions ($\mathsf{txn}:\mathfrak{q}\rightarrow\mathfrak{t}$) and (${\tau}:\mathfrak{q}\rightarrow\mathfrak{p}$) to relate a query to the transaction instance calling it and the partition it is executed on. We use an auxiliary function $\mathtt{Alive}{}{}:\mathfrak{r}\times\mathfrak{q}\rightarrow\mathbb{B}$ to capture the liveness of a record in a particular query instance. Predicates $\mathtt{vis}$, $\mathtt{ar}$, $\mathtt{ST}$, $\mathtt{WR}$, $\mathtt{RW}$ and $\mathtt{WW}$, all with the signature $\mathfrak{q}\times\mathfrak{q}\rightarrow\mathbb{B}$ are also defined to capture relations discussed in section 3. The disjunction of dependency relations $\mathtt{WW}$, $\mathtt{WR}$ and $\mathtt{RW}$ is denoted by $\mathtt{D}$.

$\varphi_{\text{{anomaly}}}$

This clause specifies that an independent serializability anomaly exists in the program, that consistes of $i$ serially executed transactions and $j$ concurrent transactions that manifest a dependency cycle of length $k$. It is also required that all serializable transactions are executed before the transactions involved in the cycle.

[TABLE]

In the above formula, $\varphi_{\text{{ser}}}(t)$ encodes the serializability of $t$ with respect to all other transaction instances and $\varphi_{\text{{order}}}(t,t^{\prime})$ enforces that all queries from $t$ are arbitrated before the queries of $t^{\prime}$. Proposition $\varphi_{\text{{cycle}}}^{k}(t_{1},...,t_{j})$ forces the solver to instantiate $k$ queries within the given transactions, such that a cycle of length $k$ exists between them, at least two of its edges are dependency relations in $\{\mathtt{WR},\mathtt{RW},\mathtt{WW}\}$ and one is an $\mathtt{ST}$ relation. The rest of the edges can be of either kind.

[TABLE]

Figure 14 presents concrete examples of dependency cycles detected by clotho. Figure 14(a) depicts a scenario where the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}query from oneRead transaction witnesses the intermediate value written by twoWrites transaction. clotho automatically determines the execution order of queries (shown as circled numbers) and values of input arguments required for manifestation of the anomaly. In this example, it suffices that all queries access the record with the same key, i.e. $\mathsf{k_{1}=k_{2}=k_{3}}$. The anomaly depicted in figure 14(b) occurs when an update from transaction $\texttt{twoWrites}^{\prime}$ is executed at a node which then becomes disconnected from the network to form partition 1. Assuming that clients can access the rest of the network on partition 2, the remaining queries need to be submitted to available nodes in the specified order in order to trigger the depicted serializability anomaly. This anomaly does not occur when network partitioning is not possible. Figures 14(c) and 14(d) respectively present the cycles detected for external and internal lost update anomaly from the example discussed in section 4.2. Predicate $\mathtt{ST^{+}}$ is defined similarly to $\mathtt{ST}$ but also requires that no data dependency exists between the predicated query instances. clotho users may optionally choose to only detect internal serializability anomalies in their programs, in which case $\mathtt{ST}$ will be replaced with $\mathtt{ST^{+}}$ in $\varphi_{\text{{cycle}}}$.

$\varphi_{\text{{context}}}$

This clause ensures that a satisfying assignment corresponds to a valid execution of database programs. It uses a collection of functions $\mathtt{init}^{f}\!:\!\mathfrak{r}\rightarrow\mathbb{Z}$ and $\mathtt{val}^{f}\!:\!\mathfrak{q}\times\mathfrak{r}\rightarrow\mathbb{Z}$ which are defined for all fields $f$ in the given schema. These functions are used to identify the initial values of records and their values at each query operation, respectively. These functions are constrained so that the values read by operations are from the initial state if the fields have not yet been written to. We make this explicit via the following formula:

[TABLE]

Next, $\varphi_{\text{{context}}}$ constrains the values of all records stored in a variable $v$ by a {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}query to satisfy the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}condition of that query.

The next set of restrictions encoded in $\varphi_{\text{{context}}}$ constrains the fields of records involved in a dependency cycle in order to make the corresponding anomaly more understandable. As a concrete example, consider the $\mathsf{twoWrites}$ transaction in figure 14(a). Without explicit constraints on the updated values in this transaction, a SMT solver could trivially instantiate parameters $\mathsf{a_{1}=a_{2}=0}$, making the dirty value read by $\mathsf{oneRead}$ transaction the same as the value read in a serializable execution of the two transactions. This could falsely appear benign to developers, so $\varphi_{\text{{context}}}$ constrains the values of records to avoid such scenarios. $\varphi_{\text{{context}}}$ forces the values of field $f$ used in queries $q_{1}$ and $q_{2}$ to be identical if there is a read dependency between them, i.e. $\mathtt{WR}(q_{1},q_{2})$ holds. Similarly, if there is an anti-dependency ($\mathtt{RW}$) or a write dependency ($\mathtt{WW}$) between two queries, the values of any fields written or read by those queries are forced to be different.

Finally, we constrain the visibility relation to capture how effects in a partition are witnessed by any later query executed in that partition, i.e. we enforce causal visibility within each partition:

[TABLE]

$\varphi_{\text{{db}}}$

This clause enforces any consistency or isolation guarantees (e.g. those from Table 1) provided by the database under test, as well as any user-defined constraints on record instances, e.g. requiring a table to be initially empty or requiring $age$ field to be always greater than 21.

\varphi_{\text{{dep\rightarrow}}}

This clause ensures that every database operation captures the effects of any query it is related to by the dependency relation $\mathtt{D}$. This is accomplished via a proposition, $\mu^{\mathtt{D}\rightarrow}_{{q,q^{\prime}}}$, which asserts that

(a) there exists a concrete record instance that both queries access and

(b) both queries are reached by the control flow of their transactions.

To this end, we first introduce function ${[\![]\!]^{\mathbb{B}}_{t}}:\phi\rightarrow\mathbb{B}$ which constructs an FOL formula corresponding a boolean expression in $\mathcal{AR}$, alongside any additional conditions that must be satisfied for a successful construction. When the input is a {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}condition of a query, a record instance (that must be checked if satisfies the condition) is also passed to the function as an extra argument, e.g. ${[\![]\!]^{\mathbb{B}}_{t,r}}$. We also assume a function $\Lambda:\mathfrak{q}\rightarrow\phi$ which returns the conjunction of all conditionals which must be satisfied in order for a query to be reached by the program’s control flow.

Figure 15 gives a rule defining a formula that enforces these conditions when an anti-dependency relation $\mathtt{RW}({q,q^{\prime}})$ is established between a {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}query ${q}$ and an {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}query ${q^{\prime}}$ which access the same field ${f}$. The transactions containing the two query are distinct. The rule enforces that there is a record which satisfies the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{WHERE}\;}clause of both queries and is perceived alive by both of them. Both queries are also forced to be reachable in their corresponding transactions. Assuming the full set of rules defining $\mu^{\mathtt{D}\rightarrow}$ for a given program, the \varphi_{\text{{dep\rightarrow}}} clause is simply defined as their conjunction:

[TABLE]

\varphi_{\text{{\rightarrowdep}}}

Examining the values of $\mathtt{ar}$, $\tau$, $\mathtt{init}$ and $\mathtt{val}$ in a satisfying assignment to $\varphi_{\mathbb{C}}$, gives us enough information to recover a concrete execution of all the queries involved in a dependency cycle. This suffices for constructing valid test cases for all the examples discussed so far, all of which consist of transactions that only include queries involved in the dependency cycle. In practice, though, we observed that for the majority of detected anomalies, transactions include operations which occur before the start of the cycle. Such operations may cause the database’s pre-cycle state to diverge from the state built by the conjunction of the clauses previously described, preventing clotho from constructing a valid test configuration that realizes the anomaly.

As a concrete example, consider the delivery transaction from the TPC-C benchmark, where records from the tables $\mathsf{n\_order}$ and $\mathsf{cust}$ are retrieved and updated. As presented in Figure 16 (left), a lost update anomaly on $\mathsf{cust}$ table may occur if two instances of delivery transaction concurrently select and update the same $\mathsf{cust}$ record (depicted by red $\mathtt{RW}$ edges). Note that, the primary key for the $\mathsf{cust}$ record involved in this anomaly, is determined based on the $\mathsf{n\_order}$ records previously retrieved during steps ① and ③.

Mapping this cycle to a non-serializable execution requires finding transaction arguments that force the retrieval of $\mathsf{n\_order}$ records at steps ① and ③ that refer to the same $\mathsf{cust}$ record. Following proposition (2), our encoding thus far constrains the values of the $\mathsf{n\_order}$ records (including values in the $alive$ field) retrieved by the queries at ① and ③ to be the same. Both queries will witness the initial state of the records, since there does not exist any $\mathtt{WR}$ pointing at them. As a result, a satisfying assignment could select the same value as the argument to both transactions, i.e. $\mathsf{a_{1}=a_{2}}$. When replaying of this execution according to the specified arbitration orders is attempted, however, the $\mathsf{n\_order}$ record retrieved at step ① is deleted at step ② and cannot be retrieved at step ③. Consequently, the execution will either terminate at step ③ or fail to form the desired dependency cycle, since two different $\mathsf{cust}$ records would be selected in steps ⑤ and ⑥.

The \varphi_{\text{{\rightarrowdep}}} clause remedies this problem by forcing $\mathtt{WR}$ edges to exist between operations that are not part of the dependency cycle. This clause forces the $\mathtt{WR}$ edge depicted by the green arrow in Figure 16 (right) to be established. When combined with the previous arbitration ordering, this prevents the operation at step ③ from selecting the same $\mathsf{n\_order}$ record as the one deleted at ② and eliminates the spurious dependency cycle between steps ⑤, ⑥, ⑦ and ⑧. A satisfying assignment must now ensure that two records exist in the $\mathsf{n\_order}$ table, both of which refer to the same $\mathsf{cust}$ record, in order to subsequently manifest a valid lost update anomaly on that record.

The clause \varphi_{\text{{\rightarrowdep}}}, using predicates $\mu^{\rightarrow\mathtt{D}}_{{q,q^{\prime}}}$, defines sufficient conditions under which a dependency relation $\mathtt{D}$ between $q$ and $q^{\prime}$ must exist. Figure 17 presents the rule that defines the predicate to force a $\mathtt{WR}$ dependency when ${q}$ is a {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}query and ${q^{\prime}}$ is an {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}such that:

(1) the update query is visible to the select query,

(2) both queries access the same alive row, and

(3) both $q$ and $q^{\prime}$ will be executed.

Using the complete set of rules for $\mu^{\rightarrow\mathtt{D}}$, we can give the full definition of the clause that forces the solver to establish dependency edges also between operations outside of a cycle:

[TABLE]

We describe refinements to this encoding scheme in the following section that allow clotho to manifest serializability anomalies for realistic database programs, as evidenced by our experimental results. These results support our contention that our encoding is sufficiently precise to map abstract executions to realizable non-serializable concrete ones.

6. Implementation

In this section, we first present our algorithm for exhaustively finding the set of anomalous execution in a given application and then discuss details related to clotho’s implementation.

6.1. Search Algorithm

The naïve algorithm described in the previous sections that iteratively queries a SMT solver to find all solutions to $\varphi_{\mathbb{C}}$ is unfortunately too inefficient to be effective in practice. In our initial experiments, we found that the SMT solver would often fail to find any satisfying assignment to $\varphi_{\mathbb{C}}$ within a reasonable amount of time. We hypothesized that the reason for this was that the solver had the flexibility to instantiate $\varphi_{\text{{anomaly}}}$ in (exponentially many) ways that did not force a cycle. Exploiting this intuition, we designed a two-stage search algorithm, presented in Figure 18, which iteratively guides the solver towards cycles via a series of successively more constrained SMT queries and then attempts to construct independent anomalies based upon those cycles.

The FindAnomalies algorithm takes as input a data schema $s$, an $\mathcal{AR}$ application $a$, a set of user-defined constraints on the database and its consistency model $\varphi_{\text{{db}}}$, and $max_{p}$, $max_{t}$, $max_{c}$, which are bounds on the space of independent anomalies, as described in section 5. The algorithm also outputs $anoms$, the set of all satisfying assignments to $\varphi_{\mathbb{C}}$. The algorithm constructs $anoms$ by repeatedly querying a SMT solver, denoted by isSAT. Satisfying assignments to $\varphi_{\mathbb{C}}$ are constructed by first iteratively finding the set of all bounded cycles using the SMT query in line 5, and then constructing independent anomalies atop of each cycle using the SMT query in line 17.

The formula in the first query (line 5) is a conjunction of four clauses quantified over $t$ variables which represent the transactions on a dependency cycle of length $c$. $\varphi_{\text{{app}}}$ is constructed before the loop begins using function EncApp and corresponds to \varphi_{\text{{dep\rightarrow}}} and \varphi_{\text{{\rightarrow dep}}} in the given program. Finally, the formula also includes a clause $\varphi_{\text{{neg}}}$, representing the negated conjunction of all previously found assignments (stored in $cycles$) so that the solver is forced to find new cycles at each iteration. The resulting satisfying assignment is stored in the $new\_cyc$ variable, which is then added to $cycles$ in line 7. Similarly, the query in line 17, encoded by function EncPath, is quantified over $p$ serially executed transactions that are intended to update the initial state of the database such that the pre-conditions for the given cycle can be satisfied.

The above steps describe how to efficiently construct a satisfying assignment to $\varphi_{\mathbb{C}}$. When experimenting with clotho , we observed that many anomalies share a similar structure, in that they all include the same sorts of dependency relations between the operations in the transactions involved in the cycle. As an example, consider the program in Figure 19 where five different anomalies (depicted using dependency edges of different colors) can form on the two transaction instance, all sharing the structure:

{\mathsf{txn1}}$${\mathsf{txn2}}$$\scriptstyle{\mathtt{WR}}$$\scriptstyle{\mathtt{RW}} ​​.

We refined our algorithm to exploit this observation by guiding the solver to find all such structurally similar anomalies once one of them is detected. It does so via an inner loop (lines 9-13) which repeats the steps discussed above, but first adds a clause, $\varphi_{\text{{stct}}}$, that forces the solver to look for cycles that are structurally similar to the one found on line 5. $\varphi_{\text{{stct}}}$ is constructed by the function EncStruct which constrains the dependency edges and transactions that the solver considers on a cycle. The inner loop repeats until all such cycles are found. When the algorithm can no longer find any new anomaly with the shared structure, it breaks the loop in line 12, and the normal execution of the outer loop is resumed. We empirically demonstrate in section 7 that this optimized procedure is more performant than one that does not include the inner loop.

6.2. Prototype

clotho222https://github.com/Kiarahmani/CLOTHO is a complete directed testing framework that combines the SAT-based approach of section 5 with a front-end compiler and test replay environment. The front-end compiler takes as input an arbitrary Java class which manipulates a database through the standard JDBC API, where each method is treated as a transaction. The compiler supports queries written in the subset of SQL given in Figure 5. Consequently, it does not allow nested queries, joins or update operations with implicit reads (e.g. {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}\mathtt{T}\;{\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SET}\;}\mathtt{f\!\!=\!f\!+\!1}). These operations are not typically supported by distributed, weakly consistent databases, and translating application logic to avoid them is a necessary and standard step of porting a database application to a distributed systems.

Our compiler is implemented as a pass on Shimple programs which are produced by the Soot static analysis framework (Vallée-Rai et al., 1999). The compiler initially performs a simple goto analysis to determine loops and conditional structures and marks all variables that are modified or read within them. It then detects all database access points, e.g. executeUpdate() function calls. For each access point, it then constructs an $\mathcal{AR}$ query that may contain typed holes. The compiler then walks backward through all assignments of variables used in the query and tries to fill the holes with $\mathcal{AR}$ values. In the case that a variable is assigned a value produced by an unknown function call, the compiler creates a new abstract variable to use in place of the variable in the original query. clotho  then uses the algorithm presented in Figure 18 to search for anomalies, using Z3 (de Moura and Bjørner, 2008) as a backend.

For each satisfying assignment found, clotho extracts a set of test configuration folders, each of which contains a database initialization file and a set of Java class files annotated with execution orders and input parameters. clotho’s test administration system uses this information to replicate anomalous executions on a concrete environment. The test environment is initialized by creating a cluster of Docker containers running replicated database instances that have been instantiated according to the test configuration. The replayer supports any distributed or centralized database that has JDBC drivers. The replayer also spawns a set of Java worker containers executing transactions from the test configuration with the specified parameters. Partitioning requirements are implemented using the open-source Blockade (2019) tool which artificially induces network partitions between replica or worker containers as required by the test configuration. Transaction schedules are enforced by a custom JDBC driver that receives requests from application workers and acts as a managed wrapper to the driver for the underlying database. Requests are blocked until the driver communicates with the central scheduling unit using the RMI functionality in Java, and are only passed on to the actual driver once the permission to proceed has been acquired.

7. Evaluation

The goal of our experimental evaluation was to investigate the following questions:

• (R1)

Is clotho effective, i.e. capable of detecting serializability anomalies in real weakly consistent database applications?

• (R2)

Is clotho practical, i.e. how many anomalies does it find and how long does it take to find them?

• (R3)

Is our FOL encoding rich enough for cycles to be mapped to unserializable executions on a real datastore?

• (R4)

How does clotho  compare to undirected or random testing at finding bugs?

Local evaluation of clotho  was carried out on a Macbook Pro with a 2.9Ghz Intel Core i5 CPU and 16GB of memory, running Java 8 and equipped with Z3 v4.5.1.

7.1. Detecting Serialization Anomalies

To answer questions (R1)-(R3), we carried out three experiments on the set of benchmarks shown in Table 2. These applications were drawn from the OLTP-Bench test suite (Difallah et al., 2013), and were designed to be representative of the heavy workloads seen by typical transaction processing and web-based applications that can also be deployed on weakly consistent data stores (Bailis et al., 2014b; Shasha and Bonnet, 2003).

A few of these benchmarks have distinctive features that deserve special mention. SEATS is an online flight search and ticketing system with both read and write intensive transactions. TATP simulates a caller location system with non-conflicting transactions that are prone to be rolled back, which complicates compilation. SmallBank simulates a financial account management system comprised of transactions that repeatedly access the same objects and therefore heavily depend on transactional isolation guarantees. Twitter and Wikipedia are inspired from the popular online services featuring micro-blogging and collaborative editing facilities. The data model and the transactional logic of Wikipedia is notably complex, where transactions manipulate a large set of backed-up tables for user-authentication and concurrent edit history management. To apply clotho to this test suite, we had to manually replace any unsupported database operations with equivalent operations, per section 6. Out of the 131 distinct queries found in these benchmarks, only 11 used {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{JOIN}\;}\!, which we replaced with other supported queries and application-level loops. While we made these edits by hand, this process has been shown to be mechanizable (Tahboub et al., 2018).

Our first experiment investigated questions (R1) and (R2) by applying clotho  to detect serializability anomalies of a fixed length in the benchmark applications. The first eight columns of Table 2 presents the results of this experiment. For each benchmark the table lists the number of transactions (#Txns) and number of tables (#Tables) it includes, the maximum bound on cycle length considered (Max. Length), the total number of anomalies found (#Anmls), unique number of structures (#Dist Strcts), the average time needed to detect a single anomaly (Avg Anlys), and the total analysis time (Total Anlys). For each benchmark, we initially limited the length of cycles to be at most 4, as all the canonical serializability anomalies, e.g. dirty reads and lost updates are of this length (Berenson et al., 1995). As section 7.2 will discuss in more detail, this bound is also sufficient to discover executions violating every correctness criteria in the specification of the TPC-C benchmark. For the two benchmarks in which no anomaly of length 4 was found, TATP and Voter, we iteratively increased the bound but were unable to find any longer anomalies. These results demonstrate that clotho can effectively detect anomalies in a reasonable amount of time on our benchmark applications, needing about an hour to analyze our most complex application.

Our next experiment tackled (R3) by using the test configuration information generated by clotho to manifest potential anomalies on the popular Cassandra database storage engine. We chose Cassandra because it only guarantees eventual consistency and provides no transactional support, allowing for all serializability anomalies to potentially occur and be caught.

Column (Auto Replay) in Table 2 lists the number of unserializable executions clotho’s test administration framework was able to exhibit from the test configurations generated by clotho, accounting for over 80% of detected anomalies. There were two main reasons this process failed for the other anomalies: underspecified loop boundaries in the encoding and the mismatch between the order of fetched rows in the encoding and at the runtime. Nevertheless, we were able to edit the generated test configurations to include enough information to exhibit an unserializable executions in all the remaining cases. In particular, by specifying the loop boundaries in the encoding and modifying the order at which initial database rows are inserted. Column (Man. Replay) lists the number of anomalies that required such interventions.

To test the effectiveness of the optimized search algorithm presented in section 6.1, we created an unoptimized variant of FindAnomalies by removing its inner do-while loop. To compare the two algorithms, we applied the unoptimized version to the three benchmarks with the most anomalies, using the total analysis of the optimized version as a timeout. Figure 20 presents the number of anomalies found by the basic version against the optimized. We can see that the optimized algorithm performs better than the unoptimized version, reporting more than double the anomalies in the same amount of time for each benchmark.

7.2. Comparison with Random Testing

Our final experiment was designed to compare clotho’s ability to find “real” bugs over automated blackbox testing. To do so, we used the TPC-C benchmark from the previous section, as it specifies a set of twelve consistency requirements (cr) that must be preserved by any sound database system. These crs give us objective notions of buggy behaviors that can arise in realistic OLTP applications, which we used to compare the relative effectiveness of clotho and random testing. The rows ($\textsc{cr\footnotesize{1}}-\textsc{cr\footnotesize{12}}$) of Table 3 list all of TPC-C consistency requirements. We have broken ”if and only if” requirements into separate rows.

In addition to crs, we identified seven new consistency requirements (ncr1-ncr7) that should also be preserved by any serializable execution of TPC-C. We identified these invariants after examining the serializability anomalies generated by clotho. These requirements capture more subtle safety properties of the application, descriptions of each of which is also shown in Table 3. As an example, ncr4 captures that a customer’s $\mathtt{c\_payment\_cnt}$ field is uniquely determined by its initial value and the number of times payment transaction has been called on that customer.

To the best of our knowledge, no random testing tool exists that can be deployed on weakly consistent databases and that can also handle our targeted benchmark suit. Thus, we were forced to develop our own tool for testing high-level application properties. Our framework builds upon Jepsen (2018), a popular open source library for testing distributed databases and systems. In addition to test automation and logging infrastructure, Jepsen is capable of systematically injecting faults (e.g. network partitions or node clock skews) during a test. Our framework extends Jepson to:

(i) manage concurrent Java clients that share a JDBC connection pool,

(ii) support enhanced logging of client-side generated information and

(iii) perform a set of customized database-wide safety checks

, each of which is required for targeted testing of application-level properties in TPC-C. We evaluated our testing framework on a fully replicated Apache Cassandra cluster running on a configurable number of t2.xlarge EC2 instances.

In order to ensure a fair comparison and mimic how applications are tested in practice, we ensured that:

(i) the initial state of the database was realistic

(ii) each transaction call’s arguments were realistic

(iii) the maximum Cassandra throughput is achieved and reported for each test.

To achieve (i) and (ii), all our tests used initial database states and transaction parameters taken directly from OLTP-Bench. In addition, we used the smallest available tables in order to make sure that the probability of conflicts was maximized. Our tool utilizes Cassandra’s sstableloader tool to automatically and efficiently bulk-load snapshots into the cluster before each test. Lastly, we made sure that none of our design decisions, e.g. the choice of Cassandra’s secondary indices would compromise throughput, resulting in an unrealistically low load that would minimize the chance for conflicts.

We repeated each experiment with 10, 50 and 100 concurrent clients, each submitting 6 transactions per second, where the frequency of transactions adhered to TPC-C’s workload requirements. These settings achieved approximately 20%, 80% and 100% of the maximum throughput we witnessed on the deployment. Each experiment was repeated with a 10 minute and then 30 minute timeout. Table 3 presents the results of these experiments, with ✓ and ✕ recording when an invariant was violated and was not violated, respectively. As the table shows, our random testing framework was only able to violate 14 out of 21 (67%) requirements.

Alternatively, we mapped each of TPC-C’s internal serializability anomalies found by clotho to the set of invariants that could be broken in scenarios arising from those anomalies. Table 4 lists the details of the internal serializability anomalies of length 4 clotho finds for TPC-C along with the set of requirements each anomaly breaks. We found that every single one of TPC-C’s consistency requirements was broken as a result of at least one serializability anomaly.

These results evidence the importance of (internal) serializability anomalies for detection of application-level bugs and showcase clotho’s ability to efficiently construct many of such anomalies that are present in a realistic database. clotho reports these anomalies as slowed-down interleaved execution steps in a form comprehensible to the developers.

8. Related Work

Serializability is a well-studied problem in the database community, and there have been a number of efforts over the years to develop static techniques to discover serializability anomalies in concurrent database systems. For example, early works by Fekete et al. (2005) and Jorwekar et al. (2007) proposed lightweight syntactic analyses to check for serializability anomalies in centralized databases, by looking for dangerous structures in the over-approximated static graph of all possible dynamic dependency conflicts. Several recent works (Bernardi and Gotsman, 2016; Cerone et al., 2015; Cerone and Gotsman, 2016; Cerone et al., 2017; Zhang et al., 2013; Warszawski and Bailis, 2017) have continued along this line, by deriving different types of problematic structures in dependency graphs that are possible under different weak consistency mechanisms, and then checking for these structures on static dependency graphs.

However, as we have also demonstrated in this paper, using just static dependency graphs yields highly imprecise representations of actual executions, and any analysis reliant on these graphs is likely to yield a large number of false positives. Indeed, recent efforts in this space (Bernardi and Gotsman, 2016; Cerone and Gotsman, 2016; Cerone et al., 2017) recognize this and propose conditions to reduce false positives for specific consistency mechanisms, but these works do not provide any automated methodology to check those conditions on actual programs. Further, application logic could prevent these harmful structures from manifesting in actual executions.

Like our work, (Kaki et al., 2018) and (Nagar and Jagannathan, 2018) also model application logic and consistency specifications using a decidable fragment of first-order logic (FOL), so that an underlying solver could automatically derive harmful structures which are possible under the given consistency specification and search for them in actual dependency graphs taking application logic into account. Although their core language supports SQL operators, they do not incorporate their techniques into a full test-and-reply environment that allows mapping anomalies identified in abstract executions to be translated to concrete inputs that can be executed in a test environment.

Brutschy et al. (2017, 2018) recently proposed an analysis technique to find serializability violations using a dependency graph-based approach. Their notion of dependencies is very conservative and is reliant upon accurate characterization of commutativity and so-called “absorption” relations between operations. Further, in a bid to make dependency characterizations local i.e. relying only on the operations involved in the dependency, they over-approximate the impact of other operations on commutativity and absorption (resulting in what they call as “far-commutativity” and “far-absorption”).

The localization strategy presented in (Brutschy et al., 2018), however, does not suit query-based models where dependences between two operations cannot be decided locally, but are reliant on other operations. For example, following their approach, there will always be a dependence between {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}queries and {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}queries that access the same field on any row, because there is always the possibility that the values from the row chosen by the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{UPDATE}\;}\! is copied into the row chosen by the {\color[rgb]{0,0,0.46875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0.46875}\mathtt{SELECT}\;}\!\!. It is unclear how to accurately express far-commutativity and far-absorption between SQL statements performing predicate read and writes. Handling such fine-grained dependencies forms an important contribution of our work.

There have been several recent proposals to reason about programs executing under weak consistency (Bailis et al., 2014a; Alvaro et al., 2011; Gotsman et al., 2016; Li et al., 2014, 2012; Balegas et al., 2015). All of them assume a system model that offers a choice between a coordination-free weak consistency level (e.g., eventual consistency (Li et al., 2012, 2014; Balegas et al., 2015; Alvaro et al., 2011; Bailis et al., 2014a)) or causal consistency (Lesani et al., 2016; Gotsman et al., 2016)). In contrast, our focus is agnostic to the particular consistency guarantees provided by the underlying storage system, and is concerned with statically identifying violations of a particular class of invariants, namely those that hold precisely when transactions exhibit serializable behavior.

(Warszawski and Bailis, 2017) presents a dynamic analysis for weak isolation that attempts to discover weak isolation anomalies from SQL log files. Their solution, while capable of identifying database attacks due to the use of incorrect isolation levels, does not consider how to help ascertain application correctness, in particular determining if applications executing on storage systems that expose guarantees weaker than serializability are actually correct.

In addition to the mentioned threads of work on database applications, in the recent years several proposals have been made addressing similar challenges in the context of shared memory concurrent programming models. For example, static (Flanagan and Qadeer, 2003) and dynamic (Flanagan et al., 2004) approaches have been introduced to determine if a particular locking policy in such programs actually enforces a desired non-interference property. Using code annotation and custom memory allocations, Rajamani et al. (2009) extended these works and proposed a framework which allows detection of violations even in the presence of ill-behaved threads that disobey the locking discipline.

In order to liberate users from writing low-level synchronization mechanisms, a whole-program analysis is offered by McCloskey et al. (2006) which compiles user-declared pessimistic atomic sets and blocks, similar to transactional memory models, into performant locking policies.

While all above approaches rely on the programmer to annotate atomic regions in their code, as a solution to an orthogonal problem, (Lu et al., 2007) presents MUVI, a static analysis tool for inferring correlated variables which must be accessed together, i.e. atomically. Several dynamic frameworks also attempted to detect potentially erroneous program executions without depending on user annotations and by looking for dangerous access patters at runtime, either for a single shared variable (Xu et al., 2005) or multiple variables (Hammer et al., 2008).

Similar to our approach, (Huang et al., 2013) and (Machado et al., 2015) rely on symbolic constraint solving to construct full, failing, multithreaded schedules that manifest concurrency bugs. However, unlike these approaches which depend on dynamic path profiling to detect conflicting operations, we analyze programs in an intermediate language which can be generated from any source language using a proper compiler. Our approach has the additional completeness guarantee that no anomaly within the given bounds is missed.

It is worth noting that all shared-memory systems assume a coherence order on writes (a property that is not enforced in weakly consistent systems) rendering any existing analysis framework for such systems inapplicable to replicated database models. Such shared-memory systems do not permit certain executions which can occur in eventually consistent data-stores while clotho’s semantics is rich enough to model these sorts of executions. As an example, consider the simple execution in Figure 22 with two concurrent programs. Assuming $x$ is initially 0, models of shared-memory systems disallow this execution, as they force the second read event in $P_{1}$ to witness $P_{0}$’s write. No such restriction applies in a weakly consistent environment, which may allow the second read to be routed to a replica that has not yet received the update from $P_{0}$’s write event.

Several violation detection mechanisms of developers’ atomicity assumption for code regions in shared memory programs have been recently introduced where certain conflicting access patterns have been deemed safe due to lack of data flow between the read and write operations involved in such conflicts (Lu et al., 2006; Lucia et al., 2010). Unlike their purely structural approach which only considers patterns within individual operations, our notion of internal serializability (to the best of our knowledge) is the first formulation of a semantic property of whole programs which takes into account actual data-dependencies between arbitrary read and write operations that form a conflicting cycle. For example, following the approach presented in these works, the external serializability anomaly presented 14(c) and the internal serializability anomaly presented in 14(d) are both considered harmful, as they include conflicting read and writes from distinct operations.

There have been a number of testing frameworks developed for distributed systems over the years. Some notable examples include MaceMC (Killian et al., 2007) a model-checker that discovers liveness bugs in distributed programs, and Jepsen (2018), a random testing tool that checks partition tolerance of NoSQL distributed database systems with varying consistency levels to enable high-availability. clotho differs from these in a number of important ways. In particular, MaceMC does not consider safety issues related to replication, while Jepsen is purely a dynamic analysis that does not leverage semantic properties of the application in searching for faulty executions. Both of these considerations have played an important role in shaping clotho’s design and implementation.

9.

Conclusions and Future Works

This paper presents clotho, an automated test generation and replaying tool for applications of weakly consistent distributed databases. clotho is backed by a precise encoding of database applications allowing it to maintain fine-grained dependency relations between complex SQL query and update operations. Notably, the encoding enforces special relationships on operations that capture concrete execution details required for an automated and complete replay of anomalous scenarios without any input from the users. We applied clotho on a suite of real-world database benchmarks where it successfully constructed and replayed a wide range of bugs, providing strong evidence for its applicability. Additionally, we compared our approach to a state-of-the-art random testing tool where clotho performed more efficiently and more reliably.

Concurrency anomalies detected by clotho can be straightforwardly fixed by using stronger database-offered consistency/isolation guarantees (e.g. by using serializable transactions). Prescribing more optimal fixes, based on the structure of dependency cycles that clotho generates is an intriguing research problem which we leave for the future works.