Data scientists are required to obtain, pre-process, and analyze data. Companies can use the insights gathered by data scientists for making important business decisions. While this task seems straightforward, there is a multitude of challenges witnessed by a career in data science.
All seems to be a tedious task, right from learning the fundamentals from data science courses to generating data science. But the major challenge lies in data cleaning for any data science operation. To be specific, 70 percent of the work of a data scientist consists of cleaning and preparing data.
An imbalanced dataset is a typical example of unbalanced data. Let us see how to use the Near-Miss Algorithm for imbalanced datasets.
What is an Imbalance Dataset?
For classification problems, imbalanced datasets are a special case where the distribution between classes is not uniform. They are usually composed of two classes: the majority or negative class and the minority class which is also known as the positive class.
Imagine, in your dataset, you have two categories to predict: Category-A and Category-B. You have a problem with imbalanced datasets when Category-A is higher than Category-B or vice versa.
So how could this be a problem?
Imagine that Category-A contains 90 records in a dataset of 100 rows and Category-B contains 10 records. You run a model for machine learning and end up with 90 percent precision. Then comes the certainty check and you get to realize that the results are not accurate. This is a common error caused by imbalanced datasets.
Near-Miss Algorithm
The Near-miss Algorithm is used to balance an imbalanced dataset and is considered as an algorithm for undersampling and is one of the most powerful ways to balance data.
The Near-Miss algorithm works by observing the class distribution, removing samples located in the higher class. Simply put, if the algorithm witnesses a case in which two near points that pertain to different classes occur, it simply excludes the one from the higher class and ensures that the balance is preserved.
Types of Near-Miss Algorithm
There are 3 main versions of the near-miss algorithm. They are listed as follows:
Type 1: In this type of Near-Miss Algorithm, unbalanced data is improvised by assessing the minimum distance (avg) between the large distribution and three farther small distribution.
Type 2: In this version, the balancing of data occurs by figuring out the distance between ‘n’ neighbors of the data points belonging to smaller classes. The largest distance obtained from this calculation is eliminated.
Type 3: This version involves the calculation of the minimum or shortest base distance between the larger distribution and three other smaller distributions close to it.
Using the Near-Miss Algorithm for an unbalanced dataset
To use the Near-Miss Algorithm for an unbalanced dataset, three major steps are followed. As a part of the first step, the distance between the points belonging to the larger class and the point belonging to the smaller class is considered.
This is done to ensure that the undersampling process is simplified. Moving to the second part, the instances belonging to the larger class are selected. While selecting these instances, it should be noted that only those who have the shortest distance are chosen. As a final step, the algorithm returns m*n instances from the larger class.
Conclusion
The choice for an appropriate method depends on the dataset and the approach as desired by the user. Near-Miss is a popular undersampling technique that is used to deal with imbalanced classes.
However, it is not the only one. Other methods of dealing with unbalanced data include random sampling, SMOTE, etc. Therefore, make sure you are thoroughly aware of the technique before proceeding with it.
