Who Comments on Rules That Change? | Federal Regulatory Tracker

Here is a study that would be easy to write and also wrong.

Take every proposed rule you can pair with its final version. Record which kinds of organizations filed comments — companies, trade associations, advocacy groups, unions, state agencies. Then record whether the rule changed on its way to finalization. Cross-tabulate and then announce which constituencies comment on rules that change. Nothing causal suggested, to be clear, but interesting.

I ran that study on 572 paired rulemakings across 31 federal agencies, and the raw numbers cooperate beautifully. Rules where a state or local government filed a comment are 13.1 percentage points more likely to show a measured change between proposal and finalization. Rules where a regulated business commented are 8.1 points more likely. Academic commenters, 9.3 points. Every single constituency I could measure points the same direction: where they show up, rules change. Hmm.

It is a tidy result and it dissolves completely upon first contact with a control variable.

What is actually being measured

FRTracker extracts obligations — actor, deontic, action, object — from the regulatory text of proposed and final rules, then matches each proposed obligation forward into the final rule. Every proposed obligation ends up labelled unchanged, modified, removed, or split. For each rulemaking I have two outcomes: whether anything changed, and what share of the proposal's obligations changed.

The universe here is a frozen one. The obligation matcher was last regenerated as a single audited run, and every rulemaking it touched left a receipt recording the inputs, the code commit, and the resulting counts. I built the cohort from those receipts rather than from the current database, and verified that the delta counts still reconcile exactly — 1,486 of 1,486 rulemakings, 28,601 matched obligations on both sides. Twenty-one rulemakings whose obligation match produced nothing at all were dropped as undefined, rather than being silently recoded as "no change." Of what remained, 572 had organizational commenter data. That is/was the universe of this back of the envelope "study." I use that term very loosely.

Commenters were classified into broad constituencies (with lots of room for edge-case argumentation) — regulated business, trade association, public-interest NGO, labor, government, professional association, academic, individual — using a classifier built for a different project months earlier. A hand audit of thirty classified organizations found twenty-nine correct. Neat.

That taxonomy has a real limit though, and it bounds what the null below can mean. About 29 percent of organizational commenters remain unclassified (8,554 organization rows in total), and on 54.7 percent of rules the majority of identified commenters are unclassified. So, to be clear, the defensible claim is not that constituency does not matter. It is that constituency does not separate at the resolution this taxonomy can currently achieve — a statement as much about my measurement precision as about the regulatory world. A sharper classifier, a better PI, or something built from filer identities rather than organization names, could find structure this one cannot see. But you go to war with the Andrew you've got, not the Andrew you want.

The variable that eats the result

The problem with "did anything change" as an outcome is that it is very nearly a function of how much there was to actually change.

The gradient is not perfect — 3–5 obligations sits above 6–10 — and the 1.000 rests on only 23 rulemakings, so read it as a strong tendency at the top of the range rather than a law or anything approaching a law.

A rulemaking that proposes more than a hundred obligations always changes at least one of them, in the universe of this study. A rulemaking that proposes two, frequently changes neither. And organized constituencies participate disproportionately in the large ones. Where a regulated business commented, the median rulemaking had 14 matched obligations; where none did, 8. Where a state or local government commented, 22 against 8. Where an academic commented, 21 against 9, on rulemakings with a median of 4,957 public comments against 31 (47 rulemakings with an academic commenter, 525 without).

So the raw finding is not "these constituencies comment on rules that change." It is "these constituencies comment on the big rules, and big rules always change something." Once I hold constant the agency, the year, the number of proposed obligations, the total comment volume, the length of the proposed regulatory text, and whether the rule was designated significant, every one of those associations stops being distinguishable from zero:

Linear probability models, agency and year fixed effects, standard errors clustered on the proposed document — descriptive.

Switch to the share of the proposal that changed — an outcome that is not mechanically inflated by pure rule size — and the raw associations were never large to begin with: the widest is state and local government at +0.059, and it too adjusts to +0.003.

And lest you think otherwise, I leaned on this hard before believing it. Dropping EPA, which is 31 percent of the universe, also changes nothing. Restricting to proposals that produced exactly one final rule does not change it. Dropping the name-pattern classifier entirely does not change it. Dropping Transportation, Treasury or Agriculture does move the government coefficient across zero — from +0.003 to between −0.001 and −0.039 — but that is what an estimate centred on zero does under resampling; it is indistinguishable from zero in every one of those cuts, which is the claim being made.

And the most obvious worry about this universe, that rules with ingested comment data differ from those without, does not bind: the two groups change at 0.766 (n=572) and 0.766 (n=893) respectively, both computed on the same zero-delta exclusion applied everywhere else in this piece.

What does vary: the agency

The null on constituencies is not a null on everything — far from it. Decomposing the variation across the 42 agency-by-constituency cells large enough to interpret:

Across those 42 equally weighted cell means — between agencies: 76.5 percent of the variance. Between constituencies: 4.5 percent.

Look at the table below and the shape is unmistakable. Read across a row and the constituencies barely separate. Read down a column and the agencies separate a great deal.

One caveat governs every cell of it: the comment-participation data are a coverage sample, not a census by any means. Absence of a constituency from a cell means it was not observed in the group, not that it did not participate.

Mean share of proposed obligations changed, by agency and constituency

Cells with fewer than eight rulemakings suppressed; those shown range from 8 to 108 rulemakings each.

At EPA, six well-populated constituencies sit inside a band 0.025 wide. At the Department of Labor, six sit inside 0.041. Meanwhile regulated business alone runs from 0.219 at HHS to 0.467 at Transportation — a 0.248 range, against a within-agency spread across constituencies of 0.041 at Labor and 0.025 at EPA once labor is set aside. Against agencies with wider internal spreads, such as Transportation and EPA overall, the gap is much narrower — roughly comparable rather than several times larger.

The classic contrast an administrative lawyer would reach for, industry versus public-interest advocates, is 0.383 against 0.377 at EPA, 0.467 against 0.551 at Transportation, 0.219 against 0.128 at HHS.

Whatever determines how much of a proposal survives to the final rule looks like a property of the agency, not of who filed — at least to me and from this vantage point. What it is about an agency that produces that difference — drafting practice, statutory constraint, litigation exposure, internal review — this data cannot say.

What this cannot tell you

It cannot tell you whether anyone got what they asked for. That is the sentence I want to leave with here, because it is the sentence the tidy version of this little foray elides.

The relationship I can observe is: a constituency participated in a rulemaking, and the rulemaking subsequently changed. The relationship that would matter for influence, capture, or agency responsiveness is: a constituency asked the agency to do X, and the agency did X. Those are different claims, and the second requires reading what commenters requested and matching it against what actually moved in the text. FRTracker does not yet observe that, so I have deliberately not interpreted any of this as evidence of influence, success, or responsiveness. More to come on all that.

For the same reason I will not characterize the composition of change. Of the matched obligations, 62.1 percent survived unchanged, 29.4 percent were removed, 6.6 percent modified, and 1.9 percent (537 obligations) were split into multiple successors. Split counts as change in the share measure used above. It is tempting to read removal as deregulatory and therefore as industry-friendly. Resist it here, because I'm not making that claim here. An obligation can vanish because industry objected, because the agency's own lawyers reworked it, because it moved to a different section, or because a court decision intervened. Direction of change without direction of request is a Rorschach test viewed through a peephole.

Work on the missing link — matching what commenters requested against what changed — is underway and has not yet produced a result I would publish or even share via email. When it does, it will be a different article with different verbs and probably lots of exclamation points.

The point

Participation in change-prone rulemakings is not demonstrated responsiveness, and the data show that the first is easy to mistake for the second. Every constituency I measured is more likely to be present when a rule changes, for the simple reason that everyone shows up for big rules and big rules almost always (if not always always) change. Control for the size and salience of what was on the table and the differences between constituencies stop being distinguishable, while the descriptive differences between agencies remain large.

If you want to know who is shaping regulatory outcomes, the participation data will not tell you. They will tell you a great deal about which agencies rewrite their own proposals — which is a real finding, and a different one.

Comment-participation data are a coverage sample, not a census; absence of a constituency means it was not observed, never that it did not participate.