Maximum Entropy

Recently I was working on a problem with Time Series.  Time Series can quickly add up to a lot of data, as you are using previous intervals to predict future intervals.  What some people do is they create a very large dataset.  Typically you will have a number of dates in your Time Series, say we have 10,000 sequential dates.  Then you will be looking at a set number of those dates at a time, say two weeks (14 days).  This will be your “window”, and you may advance it forward for each analysis.  This is common in doing things like ARMA and ARIMA, and other analysis where you are averaging over a given period.  But given a dataset, this means you have to code, introduce the logic before the data goes into your model or calculation.  What if we could just have the Dataset do what we want? We can!

What we need is a dataset where each time we make a batch request (via a Dataloader), we get a given amount of sequences, a window.  Each new request advances the window by 1 (or alternatively, we could make it advance by whatever we choose).   

Here is a Dataset Class I created to do just this.  You simply tie it to a Dataloader and you can just make requests of it to totally simply your training.  First, here is the Dataset Class:

We pair this to a Dataloader such as below:

Each batch request from the Dataloader, will get a window of seq_length.  So if we have batch_size set to 20, and our sequence length is 100, then you will end up with 20 windows of length 100, each advancing forward by one day.  The purpose of target_cols is so that you can specify which columns are targets in your prediction.  Say the first 10 columns are actual data you are predicting and the rest of the columns are just features, you can specify target_cols = 10 and when you request data from this Dataset it will give you all of the columns in the data and then just target_cols in the target.

One of the biggest bottlenecks in Deep Learning is loading data.  having fast drives and access to the data is important, especially if you are trying to saturate a GPU or multiple processors.  Pytorch has Dataloaders, which help you manage the task of getting the data into your model.  These can be fantastic to use, especially for large datasets as they are very powerful and can handle things such as shuffling of data, batching data, and even memory management.  Pytorches Dataloaders also work in parallel, so you can specify a number of “workers”, with parameter num_workers, to be loading your data.  Figuring out the correct num_workers can be difficult.  One thought is you can use the number of CPU cores you have available.  In many cases, this works well.  Sometimes it’s half that number, or one quarter that number.  There are a lot of factors such as what else the machine is doing, and the type of data you are working with.  The nice thing about Dataloaders is they can be working on loading data while your GPU is processing data.  This is one reason why loading data into CPU memory is not a bad idea………..it saves valuable GPU memory and allows your computer to be making use of the CPU and GPU simultaneously.

The best way to go about tackling this is to run a basic test.  One thing I can tell you for sure is it is painfully slow to leave num_workers set to default.  You should absolutely at least set it to something higher.  Using 0 or 1 num_workers can take say 1-2 minutes to load a batch.  Having it set correctly can get this down to a few seconds!  When you are doing a bunch of interactions in your model this really adds up and can be the primary way you speed up your training.

Here is code I used to benchmark and find my ideal num_workers:

Here is an example of the output on CIFAR-10 Data:

Obviously there are a lot of factors that can contribute to the speed in which you load data and this is just one of them.  But it is an important one.  When you have multiple GPU’s it is very important that you can feed them as fast as they can handle, and oftentimes people are falling short of this.   You can see from the above output, using at least num_workers=4 is highly beneficial.  I have had datasets where setting this parameter much higher was required for a drastic increase.  It’s always good to check!