A bit of background

A bit of background

• All natural languages change over time

A bit of background

• All natural languages change over time
• Many have suggested that language change, like other evolutionary processes, involves both directed selection as well as stochastic drift (Sapir1921, Jespersen1922, Andersen1987, Mcmahon1994, Croft2000, Blythe2012)
• Number of ways in which selective biases may influence language change (Kirby2008, Smith2013, Enfield2014, Croft2000, Haspelmath1999, Labov2011, Mcmahon1994, Zipf1949, Baxter2006, Daoust2017; +et-al.'s )

A bit of background

• All natural languages change over time
• Many have suggested that language change, like other evolutionary processes, involves both directed selection as well as stochastic drift (Sapir1921, Jespersen1922, Andersen1987, Mcmahon1994, Croft2000, Blythe2012)
• Number of ways in which selective biases may influence language change (Kirby2008, Smith2013, Enfield2014, Croft2000, Haspelmath1999, Labov2011, Mcmahon1994, Zipf1949, Baxter2006, Daoust2017; +et-al.'s )
• Signatures of selection should be inferable from the usage data (Sindi2016, Reali2010, Bentley2008, Amato2018, Kander2017; +et-al.'s)

Newberry et al. 2017, Detecting evolutionary forces in language change

• "...we quantify the strength of selection relative to stochastic drift in language evolution."

Newberry et al. 2017, Detecting evolutionary forces in language change

• "...we quantify the strength of selection relative to stochastic drift in language evolution."

• "...time series derived from large corpora of annotated texts"

  • English verb (ir)regularization; COHA
  • Frequency Increment Test (FIT)

Newberry et al. 2017, Detecting evolutionary forces in language change

• "...we quantify the strength of selection relative to stochastic drift in language evolution."

• "...time series derived from large corpora of annotated texts"

  • English verb (ir)regularization; COHA
  • Frequency Increment Test (FIT)

• "...this work provides a method for testing selective theories of language change against a null model and reveals an underappreciated role for stochasticity in language evolution."

The Frequency Increment Test (FIT)

• Feder et al. 2014 (from a family of tests of selection, cf. refs in paper)
• Series of relative variant frequencies $v_i \in (0,1)$ at time $t_i$
• Transformed into frequency increments
• $Y_i = (v_i-v_{i-1}) / \sqrt{ 2v_{i-1}(1-v_{i-1})(t_i-t_{i-1}) }$

The Frequency Increment Test (FIT)

• Feder et al. 2014 (from a family of tests of selection, cf. refs in paper)
• Series of relative variant frequencies $v_i \in (0,1)$ at time $t_i$
• Transformed into frequency increments
• $Y_i = (v_i-v_{i-1}) / \sqrt{ 2v_{i-1}(1-v_{i-1})(t_i-t_{i-1}) }$
• Rationale: under neutral evolution, the increments $v_i-v_{i-1}$ are normally distributed with a mean of 0, and variance ~ $v_{i-1}(1-v_{i-1})(t_i-t_{i-1})$ (inversely proportional to effective population size; when $0<<v_i<<1$; Gaussian approximation of the Wright-Fisher diffusion process)

The Frequency Increment Test (FIT)

• Feder et al. 2014 (from a family of tests of selection, cf. refs in paper)
• Series of relative variant frequencies $v_i \in (0,1)$ at time $t_i$
• Transformed into frequency increments
• $Y_i = (v_i-v_{i-1}) / \sqrt{ 2v_{i-1}(1-v_{i-1})(t_i-t_{i-1}) }$
• Rationale: under neutral evolution, the increments $v_i-v_{i-1}$ are normally distributed with a mean of 0, and variance ~ $v_{i-1}(1-v_{i-1})(t_i-t_{i-1})$ (inversely proportional to effective population size; when $0<<v_i<<1$; Gaussian approximation of the Wright-Fisher diffusion process)
• Test under the null hypothesis of drift ~ test that the increments are normally distributed with a mean of 0 (e.g.: one-sample $t$-test).

Problem: how to bin the data for time series

• Microbial experiments: samples that are taken at chosen intervals and resequenced
• Common approach in corpora usage: bin fixed length time segments
  • there is always a minimal time precision threshold (COHA: years)
  • but often not enough observations at fine precision
  • so: decades, years, days, minutes
  • example: daily newspaper

Problem: how to bin the data for time series

• Microbial experiments: samples that are taken at chosen intervals and resequenced
• Common approach in corpora usage: bin fixed length time segments
  • there is always a minimal time precision threshold (COHA: years)
  • but often not enough observations at fine precision
  • so: decades, years, days, minutes
  • example: daily newspaper

• Newberry et al.: use variable width quantile binning, n(bins) = log(total frequency). Assures ~same number of occurrences per bin (but bins cover different lengths of time)

$p<0.05$
$p<0.2$
$p>0.2$

Replication of Newberry et al. 2017 (36 verbs)

Replication of Newberry et al. 2017 (36 verbs)

Replication of Newberry et al. 2017 (36 verbs)

Replication of Newberry et al. 2017 (36 verbs)

Some thoughts

Some thoughts

• In broad strokes, the generalization by Newberry et al. 2017 holds - selection is indeed detected in only ~3..7 verbs (depending on binning), and drift is quite prevalent (at $\alpha=0.05$).

Some thoughts

• In broad strokes, the generalization by Newberry et al. 2017 holds - selection is indeed detected in only ~3..7 verbs (depending on binning), and drift is quite prevalent (at $\alpha=0.05$).

• However, for most individual time series, the FIT result varies between binnings (except for ~3 almost unambiguous cases)

Some thoughts

• In broad strokes, the generalization by Newberry et al. 2017 holds - selection is indeed detected in only ~3..7 verbs (depending on binning), and drift is quite prevalent (at $\alpha=0.05$).

• However, for most individual time series, the FIT result varies between binnings (except for ~3 almost unambiguous cases)

• So is it a good approach to study language change?
  Depends on the goal.

Some thoughts

• In broad strokes, the generalization by Newberry et al. 2017 holds - selection is indeed detected in only ~3..7 verbs (depending on binning), and drift is quite prevalent (at $\alpha=0.05$).

• However, for most individual time series, the FIT result varies between binnings (except for ~3 almost unambiguous cases)

• So is it a good approach to study language change?
  Depends on the goal.

• But still, what's the deal with the variation in the results...?

What's going on?

(e.g. spill, burn)

(e.g. knit)

(differences between number of bins)

(e.g., tell)

Simulating change and binning

Simulating change and binning

• Run a large number of Wright-Fisher simulations with 200 different selection coefficients $s \in [0,5]$

Simulating change and binning

• Run a large number of Wright-Fisher simulations with 200 different selection coefficients $s \in [0,5]$

• 200 generations, the "mutant" starting at 5% and 50% of the population of size 1000.

Simulating change and binning

• Run a large number of Wright-Fisher simulations with 200 different selection coefficients $s \in [0,5]$

• 200 generations, the "mutant" starting at 5% and 50% of the population of size 1000.

• For each $s$, bin the series in successively fewer number of bins
  e.g. 200 (bin length 1) -> 100 (length 2) -> 66 (length 3) etc

Simulating change and binning

• Run a large number of Wright-Fisher simulations with 200 different selection coefficients $s \in [0,5]$

• 200 generations, the "mutant" starting at 5% and 50% of the population of size 1000.

• For each $s$, bin the series in successively fewer number of bins
  e.g. 200 (bin length 1) -> 100 (length 2) -> 66 (length 3) etc

• Repeat every combination 100x for good measure

(start at 5%)

(start at 50%)

Observations

• The FIT is insensitive to binning when selection is too weak ( $s<0.01$) to be detected; beyond about $s>0.02$ (depending on the start value) sensitivity to binning increases (false negatives)

Observations

• The FIT is insensitive to binning when selection is too weak ( $s<0.01$) to be detected; beyond about $s>0.02$ (depending on the start value) sensitivity to binning increases (false negatives)
• $0.01<s<0.02$ is relatively insensitive; but also where binning can instead decrease the FIT $p$-value (false positives)

Observations

• The FIT is insensitive to binning when selection is too weak ( $s<0.01$) to be detected; beyond about $s>0.02$ (depending on the start value) sensitivity to binning increases (false negatives)
• $0.01<s<0.02$ is relatively insensitive; but also where binning can instead decrease the FIT $p$-value (false positives)
• The normality assumption is systematically violated when $s$ approaches 0.1 (unless extreme binning is applied, which increases the false negative rate)

Range of applicability of the FIT for linguistic data

• Conditions where the FIT is not reliably applicable:
  • partially completed changes, too short series
  • too few data points (sensitive to binning & absorption adjustment)
  • too long series (multiple events or processes)
  • too high selection (particularly with high binning)
  • small near-boundary fluctuations (false positives)
  • steep changes from boundary->non-boundary values
  • monotonically increasing series (normality assumption)
• Where it is:
  • weak selection, non-monotonic series away from 0/1, but window covering enough of (a single) change