Engineer's Notebook

I record daily activities, notes, thoughts, diagrams, ideas, some reference material and special insights in their own paper notebooks.  If you were going to jot something down on a scratchpad, put it in here.  Document who you were collaborating with.  Why?  When I am in stand up meetings or talking with customers I go the paper route because I can simply open up the journal and write.  Also, it is less intimidating and error prone than having a computer between you and the people you’re talking with.

Use a pen.  Erasing and re-writing is too much work and will slow you down.  Simply, draw a line through the erroneous material and continue on.

When it comes to choosing a paper journal it is a very personal decision.  Many people like the Moleskine line of journals.  Ecosystem makes similar more colorful journals of the same format.  The things to consider when choosing a journal are:

• Size.  Will I have it with me when I need it?
• Do I want one that lays flat?
• Does it utilize a closure of some kind?  Will it be a pain to open and close?
• Does it have a pocket in the back for small papers?  Do I care about that?
• Do I want plain, ruled or grid paper?
• Do I want a few pages or lots of pages?

Personally, I use a journal that is made by Miquelrius.  It is about 6 inches wide by 8 inches tall and about an inch thick.  It has rounded corners and the pages are grid ruled.  Having the grid rule makes it easy to sketch out design ideas.  Another nice thing about this page size is that you can easily “scrapbook” in index cards or ever so slightly trimmed down print outs.
  

Create a margin on the outside of every page.  Put a date in the margin when you start a new entry.  You may want to add the time as well.  Some days there isn’t much to note and I’ll continue on the page with the next day.  In that case I draw a horizontal line between days to make it more visible.  The margin can be used for icon style call outs, indicating action/todo items, work partners, work times and post activity summaries or thoughts.

It is my experience that page flags get caught on things and torn up.  Bookmarks fall out and get lost.  Therefore, I recommend using Book Darts—small metal pieces that can be used to mark your page and point to a specific line.

With so many thoughts and notes recorded in your journal you’ll want to add your contact information inside the front cover with an offer of reward if returned.

I’ve experience quite a few benefits from keeping this journal.  For example:

• Meeting action items and decisions documented and easily formally distributed.
• Look at back at project notes from years ago and see what was done and why.
• Able to reproduce work flow for prior infrequent work.
• Able to identify collaborators for project work.
• Identify when software was installed on a machine and what versions.

OneNote

For long term, shared electronic notes I utilize OneNote.  It is available on a wide variety of platforms now.  Notebooks can be local only on some platforms or synced to the cloud.  It lets you bring together your project notes into a portfolio style view.

OneNote provides numerous ways for getting information into it:

• Typed and handwritten notes (it supports inking!)
• Custom screen capture feature
• Insert a picture from a file or camera!  (Use the camera to bring in notes from your journal.)
• On Windows OneNote can OCR your inked writing, handwriting or print inside of images for search.
• Print to OneNote
• Drop in documents
• Link to web pages
• Voice notes
• Drawings by hand and assisted

Compartmentalize

I would also suggest that you not use a single journal for all your needs.  Use one for work, one for the research you do outside of work.  In addition, consider a free form journal for capturing personal thoughts, events and accomplishments.

Summary

There are many good reasons to start and maintain an Engineer's Notebook.  Get a journal today and start the habit!

Strata 2014 Retrospective

Introduction

This year I was fortunate enough to be able to attend the Strata conference put on by O’Reilly and Associates.   I started this retrospective by doing a SWOT analysis of my experience.  I then expand with some logistics info and take away research ideas.
I had high expectations for this conference given its marketing.  The last conference of this caliber that I had attended was ÜberConf.

Strengths

• The chairs in the session rooms were comfortable.
• Healthy snack options were available at breaks.
• Sponsored sessions were identified on the agenda.
• There were lots of vendors at the expo.
• Most if not all vendors had technical people on hand.
• Mobile app had maps and schedule.
• They did provide a daily print out of the session schedule and map.

Weaknesses

• The mobile app was more frustrating than useful.  It was always wanting to update.
• Being registered in the directory means you’ll start getting spam before the conference even starts.
• Navigating the website to find specific session information was difficult.  Easier to find it through Google.
• This is not an inexpensive conference to attend in terms of conference cost, travel and hotel expense.
• Full day workshop
• Late notice on software to pre-install
• Not enough AC outlets
• No tables!

Unless they address the logistics issues of the workshop environment I can not recommend attending one.

Opportunities

• From key notes, sessions and the vendors you get to learn about what tools/processes the future holds.
• Discern what tools/processes people are using now.
• Talk with other attendees about the work they are doing and the approaches they are taking to it.
• Some insights I had are:
• Many are using Python and Julia for ETL.
• R is being used for analysis.
• Data people are starting to think about discuss data patterns.  Such as the Side Kick Pattern presented by Abe Gong (@AbeGong) a Data Scientist from Jawbone.

Threats (or why wouldn’t I want to attend)

• Fundamentals are potentially better learned with targeted training.
• Attending this conference could prevent you from attending another more relevant conference.
• The target audience for this conference is narrow.  People that identify with big data, data science and business intelligence are well served by this conference.

When attending it helps to have specific questions or problems you are looking to solve.  This gives you a good context when choosing sessions and meeting with vendors.  (There are lots of vendors!)

Internet

The conference wireless was acceptable for as many people that were using it.  Internet in the hotel lobby was very good.  There is wired internet available in the hotel room for free.  I did not have an opportunity to use it.  There is also pay to use wireless available in the room.

Research

I came away from the conference with much I want to research and experiment with and people to connect.

Tools and Libraries

• scikit-learn.org
• wakari.io web based Python data analysis
• Anaconda Scientific Python development environment
• Getting IPython set up by hand is a pain—Anaconda is a must on Windows machines.
• Pandas Python data analysis library
• Microsoft PowerBI (Office 365)
• D3.js
• The R programming language
• Pentaho, Tableau and other BI tools
• Hadoop
• Aerospike

Techniques

• Adjacency Matrix
• pivot and fold operations
• hexagonal binning
• use visualization for data quality checks
• confusion matrix
• predictive modeling fundamentals
• machine learning
• The work of John Tukey (Statistics)

Speakers

• Abe Gong (@AbeGong) Jawbone Sidekick pattern
• Monica Rogati (@mrogati) Jawbone UP
• Emil Eifrem (@emileifrem) Neo4j
• Wes McKinney (@wesmckinn) Pandas creator
• Joe Hellerstein (@joe_hellerstein)
• Jeffrey Heer (@jeffrey_heer)

Joe and Jeffrey presented: Data Transformation: Skills of the Agile Data Wrangler

• Magda Balazinska

Can we make big data management easier?  Her 3 research threads are: effective, easier and cost effective.

Summary

I came away with a better appreciation of what constitutes data science, the skills needed, the tools utilized and the vendors in the different areas.  If I attend again in the future I would likely skip the workshop day.  I would do additional prep working thinking about specific questions I may have for the technical people that the vendors make available.

Getting the Enron mail database into MongoDB

Background

In order to get some experience working with Python and MongoDB I decided I would like to find a data source with a lot of free form text. This would give me experience in pulling the data into MongoDB and at a future date I’ll have a ready data source for use with learning NLTK.
Finding a large dataset that met my needs turned out to be harder than expected. I came across Hilary Mason’s link page of research-quality data sets (now a dead link, see my collection which includes as much of Hilary's as we could recover.) and found the Enron email dataset. This data set contains over 500K emails. The emails are in individual files stored in a directory structure. To me, the first step in being able to use the data is to get it into a database where I could query it.

The environment

The code was developed on an Intel Core-I7 machine with 6G of RAM and a 5400RPM hard disk. This code is I/O intensive and could have benefitted from a faster hard disk or an SSD. The host OS was Ubuntu 11.04 64bit. The tools used were Python 2.7 with pymongo and MongoDB.

The code

How to query

You can use the mongo shell to do some queries once you have loaded the data.
“use the enron_mail” database and you can do the following:
db.messages.find({ contents : /query text/i }).limit(1).skip(0);
Besides content, the document structure also includes: mailbox, subFolder and filename.
Here are some additional links with material on the shell and how to query:
http://www.mongodb.org/display/DOCS/Overview+-+The+MongoDB+Interactive+Shell http://www.mongodb.org/display/DOCS/Tutorial http://www.mongodb.org/display/DOCS/Querying http://www.mongodb.org/display/DOCS/Advanced+Queries

Analysis

Important things to note about the code:

change the MAIL_DIR_PATH variable to match your installation.

getFileContents decodes the text as being in cp1252 character set

saveToDatabase encodes the text in utf-8 for mongo compatibility

The os.walk method is key to the simplicity of this code

Here are some references on unicode:
http://docs.python.org/howto/unicode.html http://stackoverflow.com/questions/4685568/importing-file-with-unknown-encoding-from-python-into-mongodb
The full run took ~21 minutes after an initial run that probably had a bunch of files in cache. The run maxed out a single core of the CPU. The process was i/o bound with the hard drive. The full 6G of RAM was in use on the machine
My query of MonogoDB says I have 517,424 emails in the document store. It shouldn’t be too difficult to modify the code to work with your database of choice.
I hope you find this code useful and that it enables you to do some analysis with this dataset.

Addendum

Brendan McAdams @rit created a version of the code which utilizes the Python email library to produce a database with more metadata.  You can see the results of his work here: http://mongodb-enron-email.s3-website-us-east-1.amazonaws.com/ (now a dead link)