Jump to content

Advantage/Disadvantages of using spy.jobs.schedule versus something external?


Ben Hines

Recommended Posts

I have a simple DataLab notebook that identifies new signals and adds them to an asset tree.  It seems pretty simple to have this run periodically using spy.jobs.schedule.  Another option I'm considering is to extract the code from my notebook into a .py file and run it periodically on an existing Airflow infrastructure.  Some of the advantages of using my own infrastructure is include monitoring/alerting, log collection, etc.  Are there other aspects I should be considering?  For instance, there is a prominent warning in the spy documentation to be careful about scheduling jobs due to resource consumption and adverse effects on performance.  It's unclear how script complexity will affect performance.  I'm not even sure how to monitor performance of a scheduled notebook.  Any advice you can provide would be appreciated.  

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...