Chaper 6 Backward Selection results in empty set

[juejung]

There appears to be an error with the backward selection algorithm?

While

strategy = Stepwise.first_peak(design,
                               direction='forward',
                               max_terms=len(design.terms))

df_Cp = sklearn_selected(OLS,
                         strategy,
                         scoring=neg_Cp)

df_Cp.fit(Hitters, Y)
print(df_Cp.selected_state_)

works fine and results in

('Assists', 'AtBat', 'CAtBat', 'CRBI', 'CRuns', 'CWalks', 'Division', 'Hits', 'PutOuts', 'Walks')

If I change the algorithm to 'backward' I get an empty set:

strategy = Stepwise.first_peak(design,
                               direction='backward',
                               max_terms=len(design.terms))

df_Cp = sklearn_selected(OLS,
                         strategy,
                         scoring=neg_Cp)

df_Cp.fit(Hitters, Y)
print(df_Cp.selected_state_)

results in empty selections

()
()

Since I get similar outcomes with other data sets, I have a suspicion that the 'backward' method is not implemented correctly?

I also noticed that the wrapper does not warn the user when typing a keyword different from 'forward', 'backward' or 'both' and seems to simply revert to 'forward' as the standard method.

[jonathan-taylor]

Thanks, I will take a look. Indeed there should be a flag for direction here, and it was intended for "first_peak" to go "forward", as it was meant to build a model up until the AIC decreases. Therefore, my guess (without looking into it yet) is that the "backward" option starts at an empty model and sees it cannot delete anything. There would be an application of "backward" for "first_peak" but I doubt "both" would make a lot of sense (though it could be implemented). I don't think the code is meant to carry out "backward" / "first_peak". Will investigate and add suitable warnings.

…

________________________________ From: Juergen ***@***.***> Sent: Monday, March 24, 2025 9:31 AM To: intro-stat-learning/ISLP_labs ***@***.***> Cc: Subscribed ***@***.***> Subject: [intro-stat-learning/ISLP_labs] Chaper 6 Backward Selection results in empty set (Issue #40) There appears to be an error with the backward selection algorithm? While strategy = Stepwise.first_peak(design, direction='forward', max_terms=len(design.terms)) df_Cp = sklearn_selected(OLS, strategy, scoring=neg_Cp) df_Cp.fit(Hitters, Y) print(df_Cp.selected_state_) works fine and results in ('Assists', 'AtBat', 'CAtBat', 'CRBI', 'CRuns', 'CWalks', 'Division', 'Hits', 'PutOuts', 'Walks') If I change the algorithm to 'backward' I get an empty set: strategy = Stepwise.first_peak(design, direction='backward', max_terms=len(design.terms)) df_Cp = sklearn_selected(OLS, strategy, scoring=neg_Cp) df_Cp.fit(Hitters, Y) print(df_Cp.selected_state_) results in empty selections () () Since I get similar outcomes with other data sets, I have a suspicion that the 'backward' method is not implemented correctly? I also noticed that the wrapper does not warn the user when typing a keyword different from 'forward', 'backward' or 'both' and seems to simply revert to 'forward' as the standard method. — Reply to this email directly, view it on GitHub<#40>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACTM24OSN5GTRR3Y6APPIT2WAXMLAVCNFSM6AAAAABZVMD2GWVHI2DSMVQWIX3LMV43ASLTON2WKOZSHE2DGOBRGU3TCOI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [juejung]juejung created an issue (intro-stat-learning/ISLP_labs#40)<#40> There appears to be an error with the backward selection algorithm? While strategy = Stepwise.first_peak(design, direction='forward', max_terms=len(design.terms)) df_Cp = sklearn_selected(OLS, strategy, scoring=neg_Cp) df_Cp.fit(Hitters, Y) print(df_Cp.selected_state_) works fine and results in ('Assists', 'AtBat', 'CAtBat', 'CRBI', 'CRuns', 'CWalks', 'Division', 'Hits', 'PutOuts', 'Walks') If I change the algorithm to 'backward' I get an empty set: strategy = Stepwise.first_peak(design, direction='backward', max_terms=len(design.terms)) df_Cp = sklearn_selected(OLS, strategy, scoring=neg_Cp) df_Cp.fit(Hitters, Y) print(df_Cp.selected_state_) results in empty selections () () Since I get similar outcomes with other data sets, I have a suspicion that the 'backward' method is not implemented correctly? I also noticed that the wrapper does not warn the user when typing a keyword different from 'forward', 'backward' or 'both' and seems to simply revert to 'forward' as the standard method. — Reply to this email directly, view it on GitHub<#40>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACTM24OSN5GTRR3Y6APPIT2WAXMLAVCNFSM6AAAAABZVMD2GWVHI2DSMVQWIX3LMV43ASLTON2WKOZSHE2DGOBRGU3TCOI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jonathan-taylor]

This would probably be better posted as an issue about the library ISLP, but I will reference any fix to that here.

[jonathan-taylor]

Looking at this, I actually think this code runs as intended because you haven't given any argument say where to start from going backwards. I.e. there are no initial_terms so it starts with an empty model, and running backwards from an empty model there is nothing to remove.

I do agree this is perhaps not users' desired behavior and perhaps if "backward" with no initial_terms one should use self.terms_ as initial terms.

[juejung]

Ok, I can confirm that specifying something like: initial_terms=X.columns.drop('intercept') makes it work.

However, calling SequentialFeatureSelector from the sklearn.feature_selection library directly does not require specifying an initial_terms option. Something like this code snipped seems to do the job just fine:

from sklearn.feature_selection import SequentialFeatureSelector
lr_model = LinearRegression()
skl_backward = SequentialFeatureSelector(lr_model, n_features_to_select="auto", direction='backward') 
skl_backward.fit(X, Y)
selected_vars_backward = X.columns[skl_backward.get_support()]

Maybe the next iteration of the Labs for ISLP could add an example for the backwards call or the Stepwise.first_peak() with direction=backward could throw a warning that the initial model is missing, or yes, the default behavior could change to mimic the one from SquentialFeatureSelector, I assume that is the self.terms_ that you mentioned.

Thank you for looking into this.

[jonathan-taylor]

Yes, a better default is probably better. Opened a new issue on ISLP to refer to this one: intro-stat-learning/ISLP#33 [https://opengraph.githubassets.com/f9b13e6837d6fa3772be5fcd7f1a3e74fc511eda88e83eb39eb857c686f7c989/intro-stat-learning/ISLP/issues/33]<https://github.com/intro-stat-learning/ISLP/issues/33> backward feature selection · Issue #33 · intro-stat-learning/ISLP<intro-stat-learning/ISLP#33> See here: intro-stat-learning/ISLP_labs#40 github.com

…

________________________________ From: Juergen ***@***.***> Sent: Thursday, April 10, 2025 12:21 PM To: intro-stat-learning/ISLP_labs ***@***.***> Cc: Jonathan Taylor ***@***.***>; Comment ***@***.***> Subject: Re: [intro-stat-learning/ISLP_labs] Chaper 6 Backward Selection results in empty set (Issue #40) Ok, I can confirm that specifying something like: initial_terms=X.columns.drop('intercept') makes it work. However, calling SequentialFeatureSelector from the sklearn.feature_selection library directly does not require specifying an initial_terms option. Something like this code snipped seems to do the job just fine: from sklearn.feature_selection import SequentialFeatureSelector lr_model = LinearRegression() skl_backward = SequentialFeatureSelector(lr_model, n_features_to_select="auto", direction='backward') skl_backward.fit(X, Y) selected_vars_backward = X.columns[skl_backward.get_support()] Maybe the next iteration of the Labs for ISLP could add an example for the backwards call or the Stepwise.first_peak() with direction=backward could throw a warning that the initial model is missing, or yes, the default behavior could change to mimic the one from SquentialFeatureSelector, I assume that is the self.terms_ that you mentioned. Thank you for looking into this. — Reply to this email directly, view it on GitHub<#40 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACTM27Q3S76G34C6ZC56T32Y3AFPAVCNFSM6AAAAABZVMD2GWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJUHEZTCNZZGM>. You are receiving this because you commented.Message ID: ***@***.***> [https://avatars.githubusercontent.com/u/1740639?s=20&v=4]juejung left a comment (intro-stat-learning/ISLP_labs#40)<#40 (comment)> Ok, I can confirm that specifying something like: initial_terms=X.columns.drop('intercept') makes it work. However, calling SequentialFeatureSelector from the sklearn.feature_selection library directly does not require specifying an initial_terms option. Something like this code snipped seems to do the job just fine: from sklearn.feature_selection import SequentialFeatureSelector lr_model = LinearRegression() skl_backward = SequentialFeatureSelector(lr_model, n_features_to_select="auto", direction='backward') skl_backward.fit(X, Y) selected_vars_backward = X.columns[skl_backward.get_support()] Maybe the next iteration of the Labs for ISLP could add an example for the backwards call or the Stepwise.first_peak() with direction=backward could throw a warning that the initial model is missing, or yes, the default behavior could change to mimic the one from SquentialFeatureSelector, I assume that is the self.terms_ that you mentioned. Thank you for looking into this. — Reply to this email directly, view it on GitHub<#40 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACTM27Q3S76G34C6ZC56T32Y3AFPAVCNFSM6AAAAABZVMD2GWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJUHEZTCNZZGM>. You are receiving this because you commented.Message ID: ***@***.***>

[johnrushwrs]

Added some better defaults in:
#34