Sparsity and compression
We have seen in the sections before that our signals and data can often be expressed in an optimal way by changing basis. Moreover, this often results in sparse data and therefore gives us the opportunity for compression. By expressing our data in this basis most coefficients are zero or small.
The zero coefficients give rise to sparsity and the almost zero coefficients allow us to compress the data further without loosing too much information. We have seen this in the Eigendecomposition, the singular value decomposition, within regression choices, the Fourier transform, the Wavelet transform, and in other such transformations not covered here.
Recent development in mathematics have given rise to the field of compressed sensing, where not high dimensional signals are collected and transformed or compressed but we start by acquiring compressed signals and solve for the sparsest high-dimensional signal that is consistent with the collected data.
Here we will discuss sparsity and compression and give an outlook on compressed sensing. We have already seen multiple examples and we will use this chapter to contextualize these results and combine them to give rise to new ideas and further aspects.
It is worth mentioning that quite often sparsity gives rise to so called parsimonious models that avoid overfitting and remain interpretable because they have a low number of terms. This fits neatly into the discussion of Occam’s razor: the simplest explanation is in general the correct one. Simple can also mean the least coefficients and this means sparsity. Furthermore, it can also help to create more robust algorithms where outliers have less influence.
Parts of this section are based on (Brunton and Kutz 2022, chap. 3).