Regression analysis

In general, regression analysis can be understood as a set of tools that is used to estimate or establish a relationship between a dependent variable \(Y\) (also called outcome or response variable, label) and the independent variable \(X\) (also called regressor, predictors, covariates, explanatory variable or feature). If we add a regression function \(f\) and some unknown parameters \(c\) to the mix the problem can be written mathematically as \[ Y = f(X, c) \tag{1}\] where \(c\) is found by optimizing for a good fit of \(f\) to the data.

We split up the discussion along the well known topics:

• Linear Regression in 5  Linear Regression
• Non-linear Regression in 6  Non-linear Regression
  • Gradient Descent in Gradient Descent
• Optimizers in 7  Optimizers
  • Model selection/identification and over-/underfitting in Model Selection/Identification and over-/underfitting

Parts of this section are based on (Brunton and Kutz 2022, sec. 4).

Brunton, Steven L., and J. Nathan Kutz. 2022. Data-Driven Science and Engineering - Machine Learning, Dynamical Systems, and Control. 2nd ed. Cambridge: Cambridge University Press.