An Electronic Notebook To Suit My Needs

The Issue

I've been searching for the right tool for a digital laboratory notebook for ages. I've always been back at keeping paper notebooks. When you're repeating procedures with small tweaks, it doesn't make sense to refer to your recipe on page 15 in lab notebook 1. Lots of notes are written on paper towels or post-its, and taping them into a paper notebook is a hassle and not very permanent. Referring to procedures in papers is a pain - and finding anything can be a nightmare if you don't spend hours each week indexing and organizing.

This problem was exacerbated when I was a graduate student in a lab where all of the data I generated (qPCR results, western blots images from a digital device, fluorescence microscopy images) was digital and it's completely untenable now that a great deal of the work I'm doing is writing/running computer code. But keeping track of what you're doing is essential in science, whether at the bench or on the computer, and it would be nice to have an integrated way of doing that for myself and for my lab mates.

Things I've Tried

Evernote
Towards the end of my Ph.D, I started to play around with Evernote, but I already had so much crap accumulated that it wasn't worth trying to wrap a new system around it. I ended up just making a zip folder of all of my data folders (organized roughly by date). But when I joined the Dutton Lab for my Postdoc, I decided to try to go all-in, and it worked fairly well. 

Evernote is pretty simple to use, but powerful. One of my favorite features was that you could embed a file (like an excel spreadsheet) directly into a note, and then open/edit the file within the note without needing to export/edit/save/import. You can also take hand-written notes and take a picture of them and Evernote will do OCR and make your notes searchable. 

There are a couple of problems that made Evernote untenable going forward. First, I couldn't get buy-in from the rest of the lab. There's a bit of a learning curve, and others weren't willing to invest the time. When you need to collaborate on projects/data sets and others are making changes in other systems, you lose the benefit of keeping everything in one place. Second, the large file types generated when doing genomics work (sometimes >1Gb genome files) don't work that great, and more importantly the files embedded in Evernote are not (easily) accessible from the command line or other programs). Finally, and this is related to problem 2 - there's not a good way to integrate with the workflow for coding (git etc). 

File System/Google Drive/Google Docs:
The nice thing about just storing data on your hard drive and syncing with google drive is that it already comes pretty naturally to most people. You can buy all the space you need (and some universities have deals for unlimited storage), and collaboration is a breeze. For the lab notebook itself, you can use one long Doc that has an automatically-generated table of contents. Added bonus - I've recently gotten really into Paperpile as a reference/paper manager, and the integration is seamless. 

This is basically what the Dutton lab is doing now, and it works great for typical wetlab stuff. Collaboration is easy (though dealing with people that have different philosophies around data management can be tricky), and it's pretty low-impact. The main issue with this is, again, issues integrating with code. You can't use git and an auto-syncing service like google drive at the same time on the same files, and I'm not sure how it would deal with things like Jupyter notebooks. It's strangely hard to get formatted code into a google document. And I'm using a text editor for a great deal of my work, it would be nice to be able to edit my lab notebook there as well. 

Other stuff:
Other people have suggested some other stuff that I've looked into, but isn't quite right. 

  • Microsoft OneNote - similar to Evernote, and suffers from the same deficiencies (I prefer EN)
  • Jupyter notebooks - Great for code, usable with git, not so good for dealing with other types of data files (images etc)
  • Benchling - Built for labs, good sequence viewing/editing when doing small scale stuff (plasmid design etc), not great with large datasets (you've got to pay extra for space), can't edit notebook with my own text editor
  • SciNote - this product isn't available yet (though will supposedly be open source), and seems cool, though it seems like it will suffer from the same problems as Benchling.

What I'd like

I'd like a digital lab notebook that can integrate with the tools I already use for code like Atom/Jupyter. It would be easy to reference files on my hard drive, have explicit version control (like git), and share with colleagues. Unfortunately, I think some of these goals are incompatible. A hybrid of Github for code/Jupyter + a notebook in google drive with links is maybe the best I can hope for. Probably the solution is something that just knits these together a little better. Github and google drive have decent APIs, so it should be possible, I think - but I'm not sure how I would build it.