Long-running jobs

Deepnote is designed to support most data analysis workflows, many of which include long-running tasks. This page explains when your projects can be descheduled.

Simplifying a little bit, Deepnote has many machines (computers) running in the cloud (pods in a kubernetes cluster if you want the real jargon). When you start the hardware in your project, we turn on one of these machines for you and run all your computations there. Once you're done with your computations, we "deschedule" the project, essentially turning off this machine for you just like you would turn off your own laptop at the end of your workday. Unlike with your laptop though, you can view the files in your project even while it is descheduled – you only need running hardware to execute computations. We automatically start this hardware whenever you start any execution in your project (e.g. when you run a cell). After the notebook is de-scheduled, we store the outputs of cells, but the values of variables are gone (just like when you turn off your computer).

To start your hardware back up after we descheduled your project, run any cell.

How Deepnote decides when to deschedule a project

All projects get descheduled when no one is using them. We do this to keep you from using up your free hardware hours unnecessarily. The following three conditions of inactivity have to be met:

  1. The project has no user sessions - this means that you do not have Deepnote open in your web browser.

  2. The project has no running computation - this means that there is no notebook currently executing code

  3. There were no changes to the code or any execution for at least 15 minutes (customizable).

📖 Example: Bob is running a data analysis that takes 12 hours to complete. He starts the analysis and the closes Deepnote for the night. He comes back in the morning just before the analysis is complete and finds his notebook with all his variables is still live. Bob does some ad-hoc exploration and then goes to lunch with the notebook idle. After he comes back, he can still see all his outputs, but the variables and all other runtime objects are gone as the above conditions were met and the notebook was descheduled.

Furthermore, projects running on free hardware get automatically descheduled after running continuously for 24 hours. Any running computation is interrupted. In extraordinary circumstances, we also reserve the right to deschedule the project earlier. In this case, you will see a message "Your hardware was shutdown" next time you visit the project. Please note this does not apply to paid hardware – projects running on paid hardware can run as long as the owner wants.

Finally, projects running on free hardware are descheduled if their owner runs out of free quota and they don't have a paying method set up. In this case, an info message will also appear next time you visit the notebook, notifying you about running out of the free quota.

Notebook size: Keep in mind that the maximum size of a notebook, together with outputs generated by long running cell executions, is 30MB.

Custom inactivity grace period

Users using our pay-per-use hardware can customize the behavior above and prevent the machine being automatically turned off (descheduled) after 15 minutes of inactivity. This can be useful if you're unsure when your long-running job finishes and want to keep its state preserved (such as python variables or terminal outputs). You can currently set the grace period to be either 15 minutes or 24 hours.