Argument Identification in Public Comments from eRulemaking

TL;DR

This paper develops a taxonomy and machine learning system to automatically identify, classify, and determine the stance of argumentative text in public comments during the eRulemaking process, aiding agencies in managing large volumes of feedback.

Contribution

It introduces a new taxonomy of argument claims, creates a large annotated dataset, and builds a hierarchical classification model for argument identification and stance detection.

Findings

Successful automatic identification of argumentative spans

Effective classification of argument claim types

Accurate stance determination in comments

Abstract

Administrative agencies in the United States receive millions of comments each year concerning proposed agency actions during the eRulemaking process. These comments represent a diversity of arguments in support and opposition of the proposals. While agencies are required to identify and respond to substantive comments, they have struggled to keep pace with the volume of information. In this work we address the tasks of identifying argumentative text, classifying the type of argument claims employed, and determining the stance of the comment. First, we propose a taxonomy of argument claims based on an analysis of thousands of rules and millions of comments. Second, we collect and semi-automatically bootstrap annotations to create a dataset of millions of sentences with argument claim type annotation at the sentence level. Third, we build a system for automatically determining argumentative spans and claim type using our proposed taxonomy in a hierarchical classification model.

Tables

Table 1. Table 1 . Example of substantive and non-substantive comments with different supporting arguments.

Substantivity	Argument Type	Example Comment
Non-substantive	Explicit Support	This sounds good. Up front at least. It’s been a while since this country has passed any major environmental movements, and I definitely agree with one like this.
	Likely Support	Thank you, thank you for your courage and foresight. Change must happen immediately to save the planet.
Substantive	Likely Opposition	Although I approve of the proposed expansion on the definition of medical sources, I cannot support nor agree with the elimination of all deference to treating physicians.
	Likely Opposition Disputed Information	While we support certain HRSA proposals, such as those related to telemedicine and group purchasing organization (GPO) exceptions, we have serious concerns about others, which often appear to be without basis or justification.
	Explicit Opposition	After a thorough review of the CPP, we believe the wisest course of action is for EPA to withdraw the proposed rule and abandon its costly agenda to regulate carbon dioxide under the Clean Air Act. The reasons for our position are straightforward. EPA’s proposed rule:
	Overreach	Is illegal, stretching far beyond the narrow boundaries of Section 111(d) of the Clean Air Act;
	Burdensome	Imposes high costs for no meaningful benefits. Electricity will be more expensive for small business owners and entrepreneurs, which will slow economic growth, harm competitiveness, and destroy jobs. There will be no effect on global temperatures and climate change.
	Burdensome	Threatens the reliability of the nation’s bulk electric power system, which raises the prospect of blackouts and brownouts, which can in turn increase operating expenses and uncertainty, as well as reduce output and revenues.

Table 2. Table 2 . Agencies with over 30 rules in the corpus.

Agency	Number of rules
EPA	360
CMS	155
FDA	141
OSHA	112
FWS	104
FAA	98
NOAA	77
APHIS	70
USCG	65
DOT	52
ED	50
FMCSA	49
NHTSA	44
CFPB	34
HUD	34
FHWA	32
PHMSA	31

Table 3. Table 3 . Example sentences identified by the rule-based model as belonging to the respective argument type.

Argument Type	Example Sentences
Burdensome	The current rule, which requires separate filings for each legal entity, unnecessarily increases the administrative burden on CMRS contributors and USAC.
	These multiple levels of ineligibility cause an additional burden for CRNAs to have access to this technology in order to report quality measures electronically.
	This proposal ignores the major adverse economic impacts it would create, as well as the economic costs that would negate the net benefit of any effort to expand service at current total support levels.
	We are concerned, however, that the IFR will be so costly that it challenges many of our members survival due to the increased costs, paperwork burdens and administrative hurdles now imposed by the rule.
Lacks Flexibility	One deficiency to trigger re competition is overly harsh.
	180 days of service requirement is too prescriptive.
	Should the physician know about the referral … yes but there needs to be flexibility in how to proceed.
	Many growers do not have the ability to spread dry granular products or the cost is prohibitively high.
Not Sufficient Time	Most importantly, we would request additional time to collaborate with APHIS to identify further efficiencies that might be realized in lieu of a rate increase.
	Six months is not an adequate timeframe in which to begin managing episodes that will be subject to downside risk.
Conflicting Interests	This gives them an incentive to abuse their ISP relationship to give an advantage to their content over other content providers they do not own.
	A prime contractor could also gain a competitive advantage over a subcontractor in future business dealings by having obtained such sensitive information.
	Dealing with state licensing rules there are times when the state rule is in conflict with the Federal rule.
	The proposed standard conflicts with the soil fertility and crop nutrient management practice standard of the NOP regulations (7 C.F.R. 205.203.)
Disputed Information	There is no evidence to support the hypothesis that allowing cross ownership will increase with the quantity of diversity of news available in smaller markets, as hypothesized by the Chief Economist.
	The mythological notion that regulations are bad for jobs and the economy has been repeatedly debunked, but it keeps coming back.
	Specifically, such suggestions are inconsistent with Section 254(e)’s mandate that all universal service support be explicit.
Legal Challenge	The current proposal is the product of a flawed process and will likely result in the same types of policy conflicts and legal battles which have plagued wolf management in the past.
	The new 201.3(a) will inevitably invite costly litigation between growers and processors over settled issues of case law, resulting in injury to both parties.
	…adopting such provisions would lead to endless litigation between incumbents and geographic licensees.
	This proposal is fraught with issues for the nonprofit sector and creates real legal concerns.
Overreach	The proposed rule ignores Congressional intent and Supreme Court rulings, and impermissibly expands Federal jurisdiction.
	Efforts to prohibit these services represent an unnecessary Overreach
	This dual-track of enforcement violates the letter and spirit of the humane slaughter law and exacerbates problems repeatedly raised in government oversight reports.
	The proposed regulations would considerably expand federal authority into what is currently a state-level and institution-level jurisdiction.

Table 4. Table 4 . Example sentences identified by the rule-based model as belonging to the respective argument type.

Argument Type	Example
Requests Clarification	As drafted, this language is ambiguous and the body of the final rule should clearly express the EPA’s intention regarding which entity or state can claim the benefit of renewable energy dispatch.
	The proposed rule should provide examples of unwelcome conduct concerning the use of religious symbols…
	This proposed rule needs to be clarified to apply only to new construction.
	The satellite industry has been proceeding with concrete plans to launch networks using the 37 and 39 GHz bands, but satellite investment cannot continue under the regulatory uncertainty that would result from the proposed hybrid auction and license approach.
Seeks Exclusion	You must remove this clause and give religious relief organizations the freedom to truly help the children who need their help.
	Can wording be updated to allow school districts exemption from the 5 year requirement since we secure more recent updates?
	Soaps are also listed under 40 CFR180.950 that grants tolerance exemptions for both minimal risk active and inert ingredients.
	We are asking for an exemption for treated foundation seed stock to be used by licensed seed breeders when there is no untreated seed stock commercially available to them.
Lacks Clarity	There should be a clear instruction on how to measure for school readiness.
	However we find the wording in this section of the rule confusing and potentially misleading.
	USDA should clearly identify when a covered commodity is the same.
Too Broad	I believe the proposed rule is overly broad and vague as written and as a result will actually harm a lot of horses for which this practice is not used.
	Throughout the rule, broad terms are frequently used, leaving many requirements open to interpretation.
	For example, too broad a definition could effectively result in most healthcare facilities being declared a medical device manufacturer.
	The Commission also should narrow its proposal at Appendix C paragraph 270 concerning the ability of the calling party service provider to choose whether to interconnect directly or indirectly with the called party.
Too Narrow	The current rule is too narrow and outdated, and threatens the retirement security of millions of Americans.
	The Commission also should consider expanding its definition of failed and failing stations.
	This appears to be an overly restrictive criterion.
	The alternative is to cast the exemption too narrowly, as the Commission has done, in an effort to avoid inclusion of additional entities.
Explicit Support	I am writing to urge you to make the proposed rule final.
	If this proposal is not adopted, every new device will be treated as capped rental, placing negative pressure on innovation.
Likely Support	We appreciate USDA’s willingness to help state agencies maximize the audit funds.
	I support the change that will require each grantee to meet key quality indicators to receive renewal.
	I strongly urge the EPA to move forward with your proposed framework to reduce carbon pollution from existing power plants.
Explicit Opposition	Please do not go forward with the proposed rule.
	I strongly urge the agency to withdraw this thoroughly flawed rule.
	I also urge you to not to finalize these regulatory changes.
Likely Opposition	We only ask that the panel consider the negative effects of some of the statements in the proposal that would hinder quality services.
	Rural programs do not benefit from this rule at all.
	The agency should not restrict these sustainable methods of farming without data showing an actual, verified increased rate of foodborne illness.

Table 5. Table 5 . Argument hierarchy and number of occurences in the corpus.

Stance	Argument	Argument	Number
Neutral			1613085
Opposition	Explicit		19012
	Likely		89396
		Burdensome	12001
		—Lacks Flexibility	1754
		—Not Sufficient Time	3820
		Conflicting Interests	2050
		Disputed Information	20943
		Legal Challenge	1785
		Overreach	2982
		Requests Clarification	5021
		—Lacks Clarity	9848
		—Seeks Exclusion	4119
		Too Broad	478
		Too Narrow	2187
Support	Explicit		14647
	Likely		42701

Table 6. Table 6 . Claim identification and stance detection.

		LR+w2v			fastText
Setting	Type	R	P	F1	R	P	F1
bal	Arg	0.90	0.88	0.89	0.91	0.90	0.90
	Neut	0.89	0.90	0.89	0.90	0.91	0.90
	Macro-Ave	0.89	0.89	0.89	0.90	0.90	0.90
imbal	Arg	0.51	0.79	0.62	0.71	0.59	0.65
	Neut	0.97	0.89	0.92	0.94	0.97	0.95
	Macro-Ave	0.74	0.84	0.77	0.83	0.78	0.80

Table 7. (a) Claim identification: neutral vs. argumentative classification for balanced and imbalanced settings.

		LR+w2v			fastText
Setting	Type	R	P	F1	R	P	F1
bal	Arg	0.90	0.88	0.89	0.91	0.90	0.90
	Neut	0.89	0.90	0.89	0.90	0.91	0.90
	Macro-Ave	0.89	0.89	0.89	0.90	0.90	0.90
imbal	Arg	0.51	0.79	0.62	0.71	0.59	0.65
	Neut	0.97	0.89	0.92	0.94	0.97	0.95
	Macro-Ave	0.74	0.84	0.77	0.83	0.78	0.80

Table 8. (b) Stance detection for opposition vs. support.

	LR+w2v			fastText
Stance	R	P	F1	R	P	F1
Opposition	0.92	0.85	0.88	0.89	0.92	0.90
Support	0.61	0.76	0.68	0.71	0.63	0.67
Macro-Ave	0.76	0.80	0.78	0.80	0.77	0.78

Table 9. Table 7 . Claim classification of argument types.

	LR+w2v												fastText
	claim-neutral			supp v. opp			claim+neutral			claim+ensemble			claim-neutral			supp v. opp			claim+neutral
Type	R	P	F1	R	P	F1	R	P	F1	R	P	F1	R	P	F1	R	P	F1	R	P	F1
Burdensome	0.61	0.77	0.68	0.69	0.81	0.74	0.43	0.66	0.52	0.48	0.71	0.57	0.65	0.67	0.66	0.73	0.71	0.72	0.52	0.44	0.47
Conflict Int.	0.63	0.75	0.68	0.70	0.80	0.75	0.50	0.65	0.56	0.52	0.69	0.59	0.74	0.62	0.68	0.77	0.67	0.72	0.62	0.44	0.52
Disputed Info	0.65	0.72	0.68	0.70	0.75	0.72	0.29	0.52	0.37	0.33	0.52	0.40	0.66	0.65	0.66	0.70	0.67	0.69	0.44	0.31	0.36
Explicit Opp	0.59	0.66	0.62	0.64	0.70	0.67	0.37	0.60	0.46	0.43	0.64	0.51	0.66	0.56	0.61	0.71	0.62	0.66	0.52	0.42	0.47
Explicit Supp	0.62	0.73	0.67	0.73	0.83	0.77	0.44	0.66	0.53	0.48	0.72	0.58	0.71	0.66	0.69	0.81	0.75	0.78	0.64	0.51	0.56
Lacks Clarity	0.58	0.68	0.62	0.63	0.70	0.66	0.25	0.48	0.33	0.32	0.50	0.39	0.65	0.56	0.60	0.68	0.60	0.64	0.42	0.28	0.34
Lacks Flexibility	0.45	0.53	0.49	0.54	0.61	0.58	0.22	0.36	0.27	0.30	0.46	0.36	0.53	0.38	0.44	0.63	0.43	0.51	0.32	0.17	0.22
Likely Opp	0.78	0.68	0.73	0.87	0.78	0.82	0.42	0.48	0.45	0.40	0.52	0.45	0.69	0.76	0.72	0.78	0.86	0.82	0.48	0.40	0.43
Likely Support	0.62	0.52	0.57	0.94	0.90	0.92	0.28	0.41	0.33	0.29	0.44	0.35	0.57	0.57	0.57	0.92	0.94	0.93	0.38	0.29	0.33
Legal Challenge	0.69	0.78	0.73	0.67	0.78	0.72	0.40	0.61	0.48	0.42	0.70	0.53	0.72	0.56	0.63	0.70	0.57	0.63	0.46	0.32	0.38
Not Suff. Time	0.60	0.75	0.67	0.67	0.73	0.70	0.33	0.56	0.41	0.38	0.61	0.47	0.66	0.60	0.62	0.74	0.63	0.68	0.51	0.32	0.40
Overreach	0.59	0.75	0.66	0.63	0.71	0.67	0.35	0.55	0.43	0.41	0.59	0.48	0.66	0.59	0.62	0.71	0.58	0.64	0.53	0.36	0.43
Req. Clarification	0.58	0.70	0.63	0.72	0.79	0.76	0.42	0.61	0.49	0.45	0.66	0.54	0.65	0.55	0.60	0.78	0.66	0.72	0.55	0.42	0.47
Seeks Exclusion	0.68	0.86	0.76	0.78	0.90	0.84	0.47	0.75	0.58	0.46	0.79	0.58	0.75	0.71	0.73	0.82	0.77	0.79	0.57	0.42	0.48
Too Broad	0.50	0.60	0.54	0.49	0.60	0.54	0.25	0.41	0.32	0.37	0.50	0.42	0.59	0.32	0.41	0.48	0.27	0.35	0.26	0.14	0.19
Too Narrow	0.42	0.55	0.48	0.54	0.61	0.57	0.25	0.41	0.31	0.35	0.54	0.43	0.43	0.34	0.38	0.59	0.46	0.52	0.32	0.22	0.26
Neutral	-	-	-	-	-	-	0.97	0.91	0.94	0.96	0.90	0.93	-	-	-	-	-	-	0.94	0.97	0.96
Macro-Ave	0.60	0.69	0.64	0.75	0.80	0.77	0.39	0.57	0.46	0.43	0.62	0.50	0.65	0.57	0.60	0.79	0.73	0.75	0.50	0.38	0.43

Table 10. Table 8 . Most highly weighted n-grams by the LR+w2v claim identification, stance detection and claim classification models.

Argument Type	Top n-grams
Neutral	to regulations, from another, all too, pressure to, as defined, do an, system with, analyzing, and american
Argumentative	overreach, greatly benefit, unintended consequences, economic benefit, time consuming, please reconsider
Support	not cut, not delay, welcome, economic benefit, commend, appreciates, strengthening, no adverse, helps
Oppose	not support, not approve, not pass, not agree, forcing, lawsuits, not adopt, please add, exceptions
Burdensome	prohibitive, time consuming, excessive, adverse impact, unnecessary, extremely, hardships, burdens
Conflicting Interests	anti competitive, treatment, greatly benefit, conflict of, unfair, advantage, competitive, conflict, disadvantage
Disputed Information	estimates, study, facts, unintended consequences, theory, studies, claim, evidence, proof, science
Explicit Opposition	not pass, concerned, not address, oppose, urge, object, not agree, cms to, you to, not support
Explicit Support	weaken, please pass, endorse, approved, not restrict, not delay, agree with, supports, not cut, support
Lacks Clarity	forgotten, ambiguous, unsure, explain, uncertainty, specify, clarify, define, confusing, confusion
Lacks Flexibility	artificially, room, inflated, prescriptive, dictate, inflexible, high, flexible, unreasonable, flexibility
Likely Opposition	strict, violate, should consider, disappointed, consider, once again, please reconsider, forcing, potential risk
Likely Support	helps, enacted, strong, help, applaud, positive, appreciates, economic benefit, commend, economic benefits
Legal Challenge	court, suits, to court, sue, sued, legal, suit, lawsuit, litigation, lawsuits
Not Sufficient Time	delay, postpone, timely, more time, schedules, timeframe, timeline, deadline, deadlines, schedule
Overreach	over reach, too far, overstep, right, authority, permission, power, overstepping, overreaching, overreach
Requests Clarification	change, please, please change, revision, revise, revising, replace, please add, amend, please remove
Seeks Exclusion	allowing, exclusions, suspension, suspensions, exclusion, exemptions, exceptions, exemption, relief
Too Broad	open to, overly, open, be limited, overly broad, tight, to interpretation, interpretation, specific, broad
Too Narrow	firm, stricter, be expanded, limited, narrow, long overdue, strengthening, strengthen, prohibitive, restrictive

Full text

In this work we address the tasks of identifying argumentative text, classifying the type of argument claims employed, and determining the stance of the comment. First, we propose a taxonomy of argument claims based on an analysis of thousands of rules and millions of comments. Second, we collect and semi-automatically bootstrap annotations to create a dataset of millions of sentences with argument claim type annotation at the sentence level. Third, we build a system for automatically determining argumentative spans and claim type using our proposed taxonomy in a hierarchical classification model.

††journalyear: 2019††copyright: acmlicensed††conference: Seventeenth International Conference on Artificial Intelligence and Law; June 17–21, 2019; Montreal, QC, Canada††booktitle: Seventeenth International Conference on Artificial Intelligence and Law (ICAIL ’19), June 17–21, 2019, Montreal, QC, Canada††price: 15.00††doi: 10.1145/3322640.3326714††isbn: 978-1-4503-6754-7/19/06††ccs: Computing methodologies Natural language processing††ccs: Computing methodologies Machine learning††ccs: Applied computing Law

1. Introduction

Public comments submitted during the notice-and-comment portion of eRulemaking have the potential to considerably alter the course of a regulation. In recent years as the public has become more engaged in the process, the number of submissions has exponentially increased into the millions. While the Administrative Procedures Act (APA) has been interpreted to require agencies to identify and respond to substantive comments (Lubbers, 2006), agencies have struggled to keep pace with the volume of submitted information, and it has become infeasible for them to manually review and analyze all received comments (C. R. Farina and Heidt., 2012). Therefore, there is a clear need for computational tools that can assist in the drafting and reviewing process by automatically identifying pertinent arguments being made for or against the proposed regulation that can be addressed by the agency, or suggesting argument types to employ to increase the substantivity of a comment.

The task of opinion mining is to automatically process a textual document to detect subjective information, such as an opinion, sentiment or stance (Pang and Lee, 2008). Stance detection is a type of opinion mining where given a piece of text and a target entity, the goal is to automatically identify whether the author of the text holds a supporting, opposing, or neutral position with respect to the target. Stance detection has been applied to a variety of domains, including news (Ruder et al., 2018) and social media (Wei et al., 2018). In the case of comments, the authors’ target is the proposed regulation.

A related task, sentiment analysis deals with assigning a positive, negative, or neutral polarity to subjective text (Pang and Lee, 2008). Both stance detection and sentiment analysis can be carried out at the document level, sentence level, or at the sub-sentential aspect level. Aspect-based sentiment analysis (ABSA) is a more fine-grained version where different opinion polarity can be expressed with regard to different entities and their attributes, even within the same sentence (Ruder et al., 2018). Argument mining shares subtasks with sentiment analysis and stance detection, and attempts to analyze argumentation structures in text by identifying argumentative language, classifying claims and premises, and identifying argumentative relations (Lawrence et al., 2017; Stab and Gurevych, 2017; Lippi and Torroni, 2016).

While there have been previous efforts to perform stance detection, sentiment analysis, and argumentation mining on regulatory comments (Livermore et al., 2018; Kwon et al., 2007; Park et al., 2015), this domain poses a number of unique challenges.

First, as regulatory comments are submitted by a wide range of stakeholders, including affected companies, advocacy groups, and interested individuals, they represent a diversity of perspectives and arguments in support and opposition of the proposals. Commenters have different levels of sophistication and subject matter knowledge, resulting in varying comment length, depth, and writing styles, ranging from several sentences from a public submission to many dense pages from a law firm or trade association.

Second, a regulation will often have multiple sections and affect several distinct extant administrative rules and code sections. Thus, the comment may hold a nuanced position, containing multiple arguments that either support or oppose different pieces of the rule.

Third, comments may not explicitly mention the regulation or its content, yet still express a stance. For example, authors may question the rulemaking process, the agency, the legislation that created the statutory authority, or agree with another comment.

Finally, regulations will almost inevitably be promulgated. Thus, it is much more useful for the agency to understand why an author is for or against a regulation, so they can either acknowledge the points or modify the final rule, than the overall stance. A single stance is unable to provide the substantive understanding necessary for an agency to meaningfully consider the comments.

From the agencies’ perspective, comments such as the ones at the top of Table 1, without proper justifying arguments are unlikely to be treated substantively. On the other hand, comments like the ones at the bottom of Table 1, especially if they are from prominent authors, or repeated across many commenters, are more likely to result in an agency response (Livermore et al., 2018).

In this work, we are interested in the tasks of identifying argumentative text at the sentence level, classifying the type of argument claims employed, and determining the stance of the comment. To that end, our contributions are as follows. First, we propose a taxonomy of argument claims based on an analysis of thousands of rules and millions of comments. Second, we collect and semi-automatically bootstrap annotations to create a dataset of millions of sentences with argument claim type annotation at the sentence level. Third, we build a system for automatically determining argumentative spans and claim type. We show that while this is a difficult task, using our proposed taxonomy in a hierarchical classification strategy results in the best performance for argument claim identification.

2. Related Work

Computational regulatory comment analysis has received significant attention including identifying stakeholders (Arguello and Callan, 2007), clustering duplicate comments (Yang et al., 2006), topic modeling (Livermore et al., 2018), creating topic ontologies (Yang and Callan, 2008), verifiability of propositions (Park and Cardie, 2014), automatically classifying the dialogical type of relations (Lawrence et al., 2017), theoretical argumentation models (Park et al., 2015), and a rule-dependent issue hierarchy (Cardie et al., 2008).

The closest work in spirit to ours may be that of Kwon et al. (2006) and Kwon et al. (2007) who perform topic classification, claim identification and 3-way sentence-level stance detection which they refer to as claim classification, on the claims for one EPA rule. We perform a similar claim identification on a much larger scale, and our claim classification goes beyond stance into specific and generic regulatory claim types.

Park and Cardie (2014) created a dataset of under 10,000 sentences from 1,000 comments submitted on two rules to predict the verifiability of argument propositions. They remove non-argumentative propositions, which account for only 7% of their data, and train an SVM classifier using n-gram and various count features. In a similar direction,  Park et al. (2015) propose a theoretical argumentation model for assisting authors in drafting evaluable comments. Cardie et al. (2008) create a rule-dependent issue hierarchy and annotate a set of 267 comments from an FTA rule at the sentence level.

Lawrence et al. (2017) annotate a corpus of comments from one rule with an argumentative and dialogical structure. They then build models to automatically classify the dialogical type of relations between propositions.

Outside of regulatory comment analysis, opinion and argument mining have been applied to a variety of text, including movie reviews, product reviews, student essays, and legal writing (Pang and Lee, 2008; Chen et al., 2016; Stab and Gurevych, 2017; Palau and Moens, 2009).

We assume the author holds a single supporting or opposing stance at the document level. The author may never explicitly state their stance, thus it may be unobserved and need to be inferred from the text. The author may substantiate their stance through the use of various arguments employed at the sentence level, which we consider the claims, i.e. reasons justifying the stance. So in the argumentation framework, we are interested in claim identification - identifying the argumentative components, and claim classification - classifying the types of argument claims. We leave the final step of identifying premises supporting the claims and their validity or persuasiveness to future work.111Our task could be framed as a multi-aspect sentiment analysis problem, where we first determine what the pertinent aspects across proposed rules are, and use those to identify the authors sentiment toward specific aspects for a given rule and comment. For instance, if Burdensome and Flexibility were two aspects, a commenter could believe a new rule creates no additional burden or even that a burden may be removed, and that there is enough flexibility (all positive), or that the rule creates a tremendous economic burden and lacks flexibility (negative). However, given our annotation results that the much more common intent of commenters is to argue with an aspect of the proposed rule, as many more comments complain about the flexibility, clarity, cost, etc. than praise those aspects, we chose to model the problem as one of argument identification and assume either a negative or positive sentiment per aspect type. We believe this is a reasonable assumption and leave the task of automatically learning the sentiment with respect to each of the proposed aspects to future work.

3. Data

We constructed a new dataset of comment documents annotated with argument claim types at the sentence level. In this section we explain the process we used to create our dataset and semi-automatically bootstrap annotations through weak supervision.

3.1. Collection

First, we selected dockets with more than 50 and fewer than 1,000 comments submitted to regulations.gov222https://www.regulations.gov/ is cross-agency initiative hosted by the EPA as part of the eRulemaking Program providing public access to federal regulatory content. from the first date available through submission on January 1, 2019. We collected all available comment data, including associated metadata and text.333On regulations.gov authors are able to submit comment text directly through a text field or as an attachment. As attachments come in various different formats, and many are not machine readable and thus require additional OCR capabilities, in order to reduce the noisiness of the data we only include direct submissions in the corpus. We leave the inclusion of attachment to future work.

As previous work has indicated (Yang et al., 2006), there are a substantial number of form comments submitted with large portions of repeated content. In order to have a set of unique sentences, we remove duplicates by performing a fuzzy similarity match above a threshold of 95% similar at the sentence level.444This also prevents us from artificially inflating performance by having duplicate content in the training and test sets. The resulting dataset is composed of approximately 1.8 million sentences from 2095 separate dockets. Table 2 shows the distribution of comments over the top represented agencies.

3.2. Claim Taxonomy

To create the taxonomy of argument claims, three regulatory subject matter experts manually reviewed approximately 20,000 comments to identify the types of claims that are directly used to argue for or against a proposed rule. This resulted in the definition of 12 specific and 4 generic argument claims, presented below:

Burdensome: Too arduous or burdensome, either financially or administratively, or requesting to mitigate a burden.

Not Sufficient Time: Too short a time frame for either commenting or actual implementation.

Lacks Flexibility: Overly excessive, harsh, or prescriptive.

Conflicting Interests: Favorable to one party at the expense of others, creating a conflict of interest.

Disputed Information: Containing assertions, studies or premises that are flawed or have no factual basis.

Legal Challenge: Contrary to prior judicial rulings, an invitation to judicial review, or likely to increase the chance of litigation.

Overreach: Outside the agency’s scope of authority, or outside the legislative intent of the authorizing statute.

Requests Clarification: Points that need clarification, are unclear or outright demand modification.

Seeks Exclusion: Individuals or organizations receiving special treatment or exemptions from the general application of the rule.

Lacks Clarity: Needing more definition, description, or general clarification.

Too Broad: Too sweeping in nature. The rule looks to have unintended impact.

Too Narrow: Too restrictive or specific in nature.

Explicit Support (Opposition): Supporting (opposing) the rule or a portion explicitly. Usually the comment directly references the rule or regulation in a supporting (opposing) statement.

Likely Support (Opposition): Supporting (opposing) the rule or a portion implicitly.

Tables 3 and  4 present several example sentences from the corpus from each argument type. We further group the argument claims into a hierarchy, presented in Table 5.555All argument types are assumed to justify, and thus fall under, one stance, and some argument types further qualify another argument.

3.3. Annotation

In order to scalably annotate a corpus with millions of sentences, we chose a weakly supervised semi-automated approach (Ratner et al., 2017).

We start with a set of seed words and phrases identified by the subject matter experts and automatically label the data using rule-based labeling functions. The rule-based model is based on a set of rules applied using the CKY (Cocke, 1969) context-free parsing algorithm.

We employ a number of discourse, subjectivity, and other lexical cues (Kwon et al., 2007; Lawrence et al., 2017), in the labeling criteria of the rules, including semantic orientation, polarity bearing, and semantically similar word lists, such as WordNet, and direct policy mentions.

During development of the grammar, after applying the current version of the labeling functions we cluster sentences with similar subjective phrases and expand the seed set and above-mentioned cues per claim type. We then relabel the relevant clusters with the updated model and iterate.

Table 5 shows the stance and argument type hierarchy alongside the distribution of argument types in the final annotated corpus. As previous work has also shown (Cardie et al., 2008), we see that the corpus is highly imbalanced toward non-argumentative sentences, as neutral sentences make up over 87%. This is expected as many parts of comments are expressing non-argumentative statements, such as greetings, pleasantries, or other opinions that are not directly used to argue for or against a proposed rule.

Of the opinion-bearing sentences, likely support and likely opposition are the most common across all argument types, with opposition making up over 75%. This is by design, as one of the primary motivations for this work is to enable an understanding of the specific types of disagreement present in the comments.666To relate this to a quote from Anna Karenina by Leo Tolstoy: ”Happy families are all alike; every unhappy family is unhappy in its own way.” When commenters agree with the proposal, they generally express a general sense of satisfaction, while those who oppose have many reasons why.

Beyond our current task, this corpus can be utilized in a number of ways, such as describing the differences in arguments received by different agencies, correlating the rule content to the types of arguments received, and predicting if and what arguments receive attention from the agency or result in modifications to the final rule. We leave these analyses to future work.

4. Methods

This section describes our comparison of approaches to building a system for automatic claim identification and classification.

First, we employ a flat classification approach and jointly perform claim identification and classification as a multiclass problem where we classify all claim types directly and ignore the hierarchy.

Second, we separate the tasks, and first tackle the claim identification as a binary problem of identifying non-argumentative (neutral) versus argumentative sentences. Then, given a sentence has been identified as argumentative, we pose a multiclass problem of classifying the arguments into different claim types.

Finally, we split the multiclass problem above into another binary classification problem, where we identify the stance of the sentence, as opposing or supporting, and then perform a separate multiclass classification on each of the sets of supporting and opposing argument types separately. The latter results in a top-down hierarchical classification approach (Silla and Freitas, 2011), where we have separate models for claim identification, stance detection, and two separate models for the supporting and opposing argument type classification.

4.1. Model Description

We perform the binary and multiclass classification described above with a standard bag of n-grams based linear model in the form of a logistic regression model. While neural models are increasingly popular, linear models are simple, fast to train, and have been shown to have strong performance on text classification and sentiment analysis tasks (Wang and Manning, 2012; Zhang and Wallace, 2015). We use the scikit-learn (Pedregosa et al., 2011) implementation. The feature space is binary valued n-gram presence; whether a unigram or bigram occurs in the sentence. We retain the most frequent 30k n-grams. We further augment each sentence with a weighted average of its component word vectors followed by PCA (Arora et al., 2017) using pre-trained Google word2vec 300-dimensional word vectors (Mikolov et al., 2013). In addition, we train a fastText (Joulin et al., 2016) model using 50-dimensional word vectors with unigrams and bigrams. fastText represents a document as the average of its word vectors and trains a linear model on that representation. It has been shown to have perform competitively with more complex deep neural models (Le et al., 2017).

5. Experiments

For evaluation we utilize 70% of the data for training, with the remainder serving as a test and dev set, retaining the relative proportions of each argument type in each set. Due to the imbalanced nature of the data, accuracy and micro-averaged F1 can be optimized by predicting only the majority class (e.g. neutral for claim identification), so we optimize and measure results on macro-averaged F1. We use Bayesian hyperparameter optimization (Bergstra et al., 2011) implemented in hyperopt (Bergstra et al., 2013) to select the best hyperparameters for each model on the dev set. We set class weights during training to be inversely proportional to the occurrence of the class in the training data.

5.1. Results

Results for claim identification are presented in Table 6(a). We first assess the ability to learn to identify arguments in a balanced setting, with the number of instances of the neutral class downsampled. Both models achieve a macro-F1 score of around 0.90. In the imbalanced setting where neutral sentences are highly prevalent, the neutral performance remains high but argument identification drops to 0.62-0.65 F1, resulting in 0.77-0.80 macro-F1.

Given sentences that are argumentative, we first perform binary stance detection, whose results are presented in Table 6(b). This task is also imbalanced, with more opposition examples than support. This likely explains why we perform better on opposition, with 0.89-0.90 F1.

Alterntively Table 7 (claim-neutral) shows results for performing multiclass claim classification on all - oppose and support - argument types together. The overall performance is 0.60-0.64 F1, with likely opposition, legal challenge, and seeks exclusion performing the best, while lacks flexibility, too narrow, and too broad are worst.

Given a known stance for the sentence, Table 7 (supp v. opp) shows results for performing multiclass claim classification for oppose and support argument types separately. Somewhat surprisingly we see stronger overall performance on support than oppose, with a macro-F1 of 0.85 for support and 0.65-0.70 for oppose. On an argument level, likely support and likely opposition both do well, while too broad, too narrow, and lacks flexibility perform the worst.

In all of the above except claim identification, we assume we know which sentences are argumentative, or what their stance is. However, in a real-world setting we do not. Table 7 (claim+neutral) presents performance for both argumentative and neutral sentences. As expected, we achieve a significantly lower macro-F1 as the arguments’ classification performance drops, while the neutral class remains in the 0.90s. Least affected are conflicting interests, burdensome, and requests clarification.

Finally, taking advantage of the proposed hierarchical structure of the argument taxonomy, we train the claim identification, stance detection, and separate stance-based claim classification models as before, where we have knowledge of the true argumentative sentences and stance, and augment the claim+neutral data with each of their output probabilities. We use this augmented data to build an ensemble claim classification model that in addition to the original feature space has the predicted probabilities of each of the above models as input features. Performance for the ensemble model is shown in Table 7 (claim+ensemble). This model achieves a macro-F1 improvement of 4% over claim+neutral to 0.5.

5.2. Analysis

To validate whether the semi-automated corpus creation was able to successfully identify the argumentative sentences, and to further understand the ability of the models to learn the argument types and stance, we examine the top weighted n-grams from three models.

First, the top-weighted features from the claim identification model at the top of Table 8 for the neutral class have little to do with argumentation, while the features for the argumentative class capture many argumentative bigrams, such as economic benefit, unintended consequences, and adversely affect. Second, looking at stance in Table 8, most of the supporting n-grams express positivity: no adverse, hopeful, and not delay, while the opposing n-grams explicitly disapprove: not support, not approve, and please reject. It is interesting to note that in both cases our model learns negated n-grams, and correctly learns that no adverse is supporting, while not support is opposing.

Finally, for claim classification we identify n-grams highly indicative of the argument type, such as science and estimates for disputed information, lawsuit and litigation for legal challenge, artificially and unreasonable for lacks flexibility, and postpone and deadline for not sufficient time. This further shows that our corpus creation method was able to capture meaningful distinction between argumentative and non-argumentative sentences, and identify sentences associated with the argument claim types.

6. Conclusions

In this work we addressed the related tasks of identifying argumentative text at the sentence level, classifying the type of argument claims employed, and determining the stance of the comment.

We proposed a taxonomy of argument claims based on an analysis of millions of comments that is broadly applicable across different rule types. We collected and semi-automatically bootstrapped annotations to create a dataset of millions of sentences with argument claim type annotation at the sentence level, and showed that the corpus creation process was successfully able to capture argumentative sentence and our specific proposed claim types. Finally, we presented several ways for automatically determining argumentative spans and claim types, showing the relative performance of each across the different claims. Our results show that while argument classification is a difficult task, our proposed taxonomy can be used to automatically identify and classify claims, with a hierarchical classification strategy achieving the best performance.