Backing & Hacking

    • Joining RubyTogether

      At Kickstarter, we've built our platform on top of Ruby on Rails. It's the core technology we use, and we're very proud of the platform we've built with it. We're also contributors to the Ruby and Rails communities, both in terms of open-source contributions and engagement via conferences and talks. 

      This week, we're happy to announce we've also become members of RubyTogether. RubyTogether is dedicated to funding and helping the awesome volunteers who maintain the glue of the Ruby ecosystem: tools like Bundler, RubyGems, and the like.

      We're thrilled to be giving more back to the Ruby community, and we strongly encourage other large Ruby or Rails-based platforms to become members, too.

      Leave a comment
    • A/B Test Reporting in Looker

      One of the Data team’s priorities this year has been improving the Kickstarter A/B testing process. To this end, I’ve been been focused on making it easier to set up, run, and analyze experiments. This will make it more likely we'll use data from experiments to inform product design.

      Until recently, we monitored A/B tests in an ad hoc way. We use our event tracking infrastructure to log A/B test data, so while a test was running, a Product Manager or Data Analyst watched the number of users in the experiment's event stream until it reached the required sample size. At that point, we ran the numbers through an R script and sent out the results.

      Enter Looker

      Kickstarter recently adopted a business intelligence tool called Looker to support data reporting and ad hoc analysis. Looker connects directly to our Redshift cluster, which is where we store the raw event data from our A/B tests. This made me wonder whether we could use Looker to monitor experiments and report results.

      One feature of Looker we like is the ability to save and schedule queries, with the results delivered via email. If we could find a way to analyze A/B tests via SQL, then Looker could handle the rest.

      Back to School

      How can we do statistics in SQL without access to probability distributions? There are methods for generating normally distributed data in SQL, but this approach seems like overkill. We don't need to recreate a standard normal distribution on the fly. The values don't change.

      My aha moment was remembering my old statistics textbooks with look-up tables in the back. By adding look-up tables for probability distributions to Redshift, we can get good approximations of power, p-values, and confidence intervals for the typical A/B tests we run. Although this means we're simulating a continuous distribution with a discrete one, we don't rely exclusively on p-values to interpret our tests, so a difference of a few thousandths of a point won't make much difference.

      The Nitty Gritty

      As an example, I'm going to use a common type of test we run — a hypothesis test of the difference of two proportions. (If you'd like to learn more about the statistics behind this test, this is a good place to start).

      To make this concrete, let's say we're testing a new design of the Discover page, and we want to know whether it affects the number of users clicking through to project pages.

      To generate a test statistic for this type of test, we need a standard normal distribution. I generated a set of z-scores and their probabilities in R and loaded this into Redshift as standard_normal_distribution.

      The table looks something like this:

      z_score probability
      0 0.5
      0.0000009999999992516 0.50000039894228
      0.00000199999999939138 0.500000797884561
      0.00000299999999953116 0.500001196826841
      0.00000399999999967093 0.500001595769122
      0.00000499999999981071 0.500001994711402
      0.00000599999999995049 0.500002393653682
      0.00000700000000009027 0.500002792595963
      0.00000799999999934187 0.500003191538243
      0.00000899999999948164 0.500003590480523
      0.00000999999999962142 0.500003989422804
      0.0000109999999997612 0.500004388365084
      0.000011999999999901 0.500004787307365
      0.0000130000000000408 0.500005186249645
      0.0000139999999992924 0.500005585191925

      Now let's say we've already calculated the results of our experiment for two groups: control and experimental. For each group, we have the number of unique users [inline_math]n[/inline_math] who visited the Discover page, the number of unique users [inline_math]x[/inline_math] who clicked through to a project page, and the proportion [inline_math]p = x / n[/inline_math]. This can all be done with a query.

      In the sections below, I'll use the output of that query to calculate the sample proporution, standard error, and other sample statistics using subqueries called common table expressions (CTEs). If you aren't familiar with this flavor of SQL syntax, you can think of CTEs as forming temporary tables that can be used in subsequent parts of the query.

      Using a CTE, we calculate [inline_math]\hat{p}[/inline_math], the pooled proportion under the null hypothesis:
      ... ), p_hat AS (
        SELECT
          (control.x + experimental.x) / (control.n + experimental.n) AS p
        FROM
          control, experimental
      ), ...
      

      Next we calculate the pooled standard error under the null hypothesis:

      ... ), se_pooled AS (
        SELECT
          SQRT((p_hat.p * (1 - p_hat.p)) * (1 / control.n + 1 / experimental.n)) AS se
        FROM
          control, experimental, p_hat
      ), ...
      

      This allows us to calculate an exact z-score from the data:

      ... ), z_exact AS (
        SELECT
          ABS((control.p - experimental.p) / se_pooled.se) AS z
        FROM
          control, experimental, se_pooled
      ), ...
      

      Then we find the nearest z-score in our standard normal look-up table and use that to calculate a p-value:

      ... ), z_nearest AS (
        SELECT
          standard_normal_distribution.z_score AS z_score
        FROM
          standard_normal_distribution, z_exact
        ORDER BY ABS(standard_normal_distribution.z_score - z_exact.z) ASC
        LIMIT 1
      ), p_value AS (
        SELECT
          (1 - standard_normal_distribution.probability) * 2 AS p
        FROM
          z_nearest
        INNER JOIN standard_normal_distribution ON z_nearest.z_score = standard_normal_distribution.z_score
      ), ...
      

      Having a p-value is a good start, but we also want to generate confidence intervals for the test. While we're at it, we'd also like to conduct a power analysis so the test results only display when we've reached the minimum sample size.

      To do that properly, we need some details about the test design: the significance level, the power, and the minimum change to detect between the two variants. These are all added to the query using Looker's templated filters, which take user input and add them as parameters.

      Unfortunately, Looker cannot simply add an arbitrary value (e.g. 0.05) to any part of a query. To get around this, I filter a table column with user input and then use the resulting value.

      For example, in the following section, the query takes user input as significance_level, matches it against the probability column of the standard_normal_distribution table (after some rounding to ensure a match), and saves that value as alpha:

      ... ), significance_level AS (
        SELECT
          ROUND(probability, 3) AS alpha
        FROM
          standard_normal_distribution
        WHERE
          {% condition significance_level %} ROUND(probability, 3) {% endcondition %}
        LIMIT 1
      ), ...
      

      Note Looker's syntax for what it calls templated filters:

      WHERE
        {% condition significance_level %} ROUND(probability, 3) {% endcondition %}
      

      If the user input is 0.05 for the significance_level filter, Looker converts this to:

      WHERE
        ROUND(probability, 3) = 0.05
      

      See the appendix below for the entire query.

      Admittedly, doing all this in SQL is kind of preposterous, but it means that we can add it to Looker as the engine of an A/B Test Dashboard. The dashboard abstracts away all the calculations and presents a clean UI for taking user input on the parameters of the test design, allowing people without any special engineering or data expertise to use it. Now that it's built into Looker, it's part of our larger data reporting infrastructure.

      Filters on the dashboard take user input on details about the test design
      Filters on the dashboard take user input on details about the test design

      After taking input about the test design, the dashboard calculates the number of users in each variant, their conversion rates, and the minimum sample size. If the sample size has been met, the dashboard also outputs a p-value and confidence interval for the test. The dashboard can be scheduled to run daily, and we can even set it up to email only when there are results to report.

      Emailed results
      Emailed results

      Now when we implement a new A/B test, we add it to Looker so we can get daily status emails, including statistical results when the test is complete. This can be done by someone on the Product or Engineering teams, freeing up Data team resources to focus on designing experiments well and running more complex tests.

      The Results

      This kind of dashboard pushes Looker to its limits, so naturally there are some drawbacks to doing A/B test reporting this way. It separates the implementation of the test from the implementation of the reporting, so there is some duplicated effort. Furthermore, it only works for specific types of tests where the math can be handled by a SQL query and a static probability distribution.

      On the other hand, we're happy that Looker is flexible enough to allow us to prototype internal data tools. The A/B Test Dashboard has automated what was a very manual process before, and it has reduced the dependency on the Data team for monitoring and reporting the results of common types of tests. This all means we can run more experiments to create a better experience for our users.

      Find this interesting?

      If this kind of thing gets data juices flowing, you should know we're hiring for a Data Scientist! Head over to the job description to learn more.

      Appendix

      Our query in full:
      WITH control AS (
        -- count x and n as floats
      ), experimental AS (
        -- count x and n as floats
      ), p_hat AS (
        SELECT
          (control.x + experimental.x) / (control.n + experimental.n) AS p
        FROM
          control, experimental
      ), se_pooled AS (
        SELECT
          SQRT((p_hat.p * (1 - p_hat.p)) * (1 / control.n + 1 / experimental.n)) AS se
        FROM
          control, experimental, p_hat
      ), z_exact AS (
        SELECT
          ABS((control.p - experimental.p) / se_pooled.se) AS z
        FROM
          control, experimental, se_pooled
      ), z_nearest AS (
        SELECT
          standard_normal_distribution.z_score AS z_score
        FROM
          standard_normal_distribution, z_exact
        ORDER BY ABS(standard_normal_distribution.z_score - z_exact.z) ASC
        LIMIT 1
      ), p_value AS (
        SELECT
          (1 - standard_normal_distribution.probability) * 2 AS p
        FROM
          z_nearest
        INNER JOIN standard_normal_distribution ON z_nearest.z_score = standard_normal_distribution.z_score
      ), se_unpooled AS (
        SELECT
          SQRT(((control.p * (1 - control.p)) / control.n) + ((experimental.p * (1 - experimental.p)) / experimental.n)) AS se
        FROM
          control, experimental
      ), significance_level AS (
        SELECT
          ROUND(probability, 3) AS alpha
        FROM
          standard_normal_distribution
        WHERE
          {% condition significance_level %} ROUND(probability, 3) {% endcondition %}
        LIMIT 1
      ), power AS (
        SELECT
          ROUND(probability, 3) AS beta
        FROM
          standard_normal_distribution
        WHERE
          {% condition power %} ROUND(probability, 3) {% endcondition %}
        LIMIT 1
      ), change_to_detect AS (
        SELECT
          ROUND(probability, 3) AS change_in_proportion
        FROM
          standard_normal_distribution
        WHERE
          {% condition minimum_change_to_detect %} ROUND(probability, 3) {% endcondition %}
        LIMIT 1
      ), z_alpha AS (
        SELECT
          standard_normal_distribution.z_score AS z
        FROM
          standard_normal_distribution, significance_level
        WHERE
          ROUND(standard_normal_distribution.probability, 3) = ROUND(1 - alpha / 2, 3)
        ORDER BY ABS(standard_normal_distribution.probability - (1 - alpha / 2)) ASC
        LIMIT 1
      ), z_beta AS (
        SELECT
          standard_normal_distribution.z_score AS z
        FROM
          standard_normal_distribution, power
        WHERE
          ROUND(standard_normal_distribution.probability, 3) = ROUND(beta, 3)
        ORDER BY ABS(standard_normal_distribution.probability - beta) ASC
        LIMIT 1
      ), confidence_interval AS (
        SELECT
          (experimental.p - control.p) - (z_alpha.z * se_unpooled.se) AS lower,
          (experimental.p - control.p) + (z_alpha.z * se_unpooled.se) AS upper
        FROM
          control, experimental, se_unpooled, z_alpha
      ), proportions AS (
        SELECT
          control.p AS p1,
          (control.p * change_in_proportion) + control.p AS p2
        FROM
          control, change_to_detect
      ), standard_errors AS (
        SELECT
          SQRT(2 * ((p1 + p2) / 2.0) * (1 - ((p1 + p2) / 2.0))) AS se1,
          SQRT((p1 * (1 - p1)) + (p2 * (1 - p2))) AS se2
        FROM
          proportions
      ), minimum_sample_size AS (
        SELECT
          (((z_alpha.z * se1) + (z_beta.z * se2))^2) / (p2 - p1)^2 AS n
        FROM
          z_alpha, z_beta, proportions, standard_errors
      )
      SELECT
        control.n AS step1_control,
        control.x AS step2_control,
        control.p AS rate_control,
        experimental.n AS step1_experimental,
        experimental.x AS step2_experimental,
        experimental.p AS rate_experimental,
        CASE WHEN control.n >= minimum_sample_size.n AND experimental.n >= minimum_sample_size.n 
             THEN confidence_interval.lower / control.p
             ELSE NULL
             END AS lower_confidence_interval,
        CASE WHEN control.n >= minimum_sample_size.n AND experimental.n >= minimum_sample_size.n 
             THEN (experimental.p - control.p) / control.p
             ELSE NULL
             END AS relative_change_in_rates,
        CASE WHEN control.n >= minimum_sample_size.n AND experimental.n >= minimum_sample_size.n 
             THEN confidence_interval.upper / control.p
             ELSE NULL
             END AS upper_confidence_interval,
        CASE WHEN control.n >= minimum_sample_size.n AND experimental.n >= minimum_sample_size.n
             THEN p_value.p
             ELSE NULL
             END AS p_value,
        minimum_sample_size.n AS minimum_sample_size
      FROM
        control, experimental, p_value, confidence_interval, minimum_sample_size;
      
      3 comments
    • We're going to Full Stack Fest

      Being an engineer at Kickstarter isn’t just about writing and deploying code, iterating on product features, and brainstorming new ideas. It’s also about being a part of the larger developer community: hosting meet-ups, sharing information, and finding ways to both learn from and support each other.

      One way we do that is by attending engineering conferences — and we’ve been busy this season! Here’s a little snapshot of what we’ve been up to: 

      Next up, we’ll be at Full Stack Fest in Barcelona. The weeklong programming event features a host of cool workshops and talks, plus MC Liz Abinante (@feministy). We’re also super excited that Full Stack is introducing live-captioning this year in an effort to make the conference more accessible. (On that note, did you know we launched our own subtitles and captions feature on Kickstarter recently?)

      If you want to learn more about Full Stack Fest, what they’re all about, and their take on building an inclusive conference, check out their website. Maybe we’ll see you in Barcelona!

      (Psst! Full Stack is still looking for sponsors. Full disclosure: we’re totally helping sponsor the live-captioning! ;)

      11 comments
    • Beyond Coding: A Summer Curriculum for Emerging Software Developers

      With nearly five open jobs for every available software developer, the need for qualified technical talent is higher than ever. Here in New York City alone, there are 13,000 firms seeking workers with highly sought-after tech skills: web development, mobile development, user-interface design, and more.

      So when Mayor Bill de Blasio called companies to action in support of the city’s Tech Talent Pipeline efforts, we teamed up with five other NYC-based companies — Crest CC, Foursquare, Tumblr, Trello, and Stack Overflow — to find ways to support emerging developers as they join the local tech ecosystem.

      Together, we’re introducing Beyond Coding, a new, free summer program that aims to equip emerging computer programmers with the skills to help them succeed at their first coding jobs. The goal? Provide New Yorkers who have a passion for technology with access to the mentoring, training, and support they need to really grow as developers. The curriculum is designed to address areas where junior-level technical talent might need an extra boost: it’ll cover professional networking, effective strategies for communicating technical ideas to a variety of audiences, how to prepare for an interview, and ways to gain programming knowledge outside the classroom. 

      Beyond Coding will be open to anybody in the NYC area who is currently looking for a job as a software developer (or something related), and has experience and knowledge of programming — but doesn’t have access to tools, resources, or a professional network of support. Eligible students will receive a formal certification of their course completion at the culmination of the 10-week program, and will be introduced to tech companies in New York City who are actively hiring junior-level developers. 

      Any students interested in participating in the Beyond Coding program this summer can register at beyondcoding.io. Employers interested in attending the hiring fair with certified students can email employers@beyondcoding.io for more information.

      1 comment
    • Meetup on April 8th: Open Source and Testing @ Kickstarter

      Please join us on Wednesday, April 8 at 6pm for a tour of two things we love at Kickstarter: open source and testing! RSVP via Eventbrite here.

      Light snacks and beverages will be provided — we're looking forward to a great night of tech chat!

      We'll be focusing on three topics:

      Kumquat: Rendering Graphs and Data from R into Rails (Fred Benenson – Head of Data)

      At Kickstarter our Data team works extensively in R, using Hadley Wickham’s world-class R package ggplot2 to generate data visualizations.

      Last year, we began developing internal reports using knitr, a tool to render R output and images into HTML and PDFs. The reports looked great, but we wanted a way to both automate them and also integrate them within the Kickstarter application.

      That’s why we built Kumquat: a Rails Engine designed to help integrate RMarkdown (and therefore anything R can produce) into Rails.

      I’ll go into more detail about how this works in practice and show some examples of how we’re using Kumquat at Kickstarter.

      Rack::Attack: Protect your app with this one weird gem! (Aaron Suggs – Engineering Lead)

      Modern web apps face problems with abusive requests like misbehaving users, malicious hackers, and naive scrapers. Too often, they drain developer productivity and happiness.

      Rack::Attack is Ruby Rack middleware to easily throttle abusive requests.

      At Kickstarter, we built it to keep our site fast and reliable with little effort. Learn how Rack::Attack works through examples from kickstarter.com. Spend less time dealing with bad apples, and more time on the fun stuff.

      Testing Is Fun Again (Rebecca Sliter – Engineer)

      Writing automated tests for a greenfield application is great. You define the behavior, code up the feature, and the tests pass!

      Testing becomes more challenging as an application grows. Updating behavior and corresponding test coverage can be confusing. Adding new tests can cause the test suite to run more slowly. These issues can be attributed to test architecture design. I'll provide examples of different test architectures to show how even big applications can have fun, fast tests.

      The Details

      Wednesday, April 8th
      6pm - 8:30pm
      58 Kent Street, Brooklyn, NY
      RSVP via Eventbrite here

      Spaces are limited and available on a first come first served basis.

      Note: our headquarters are located in Greenpoint, Brooklyn at 58 Kent STREET (at Franklin).

      There is a nearby Kent AVENUE, so please make sure to put in the correct address when heading here!

      The closest subway is the G train stop at Greenpoint Avenue. If you're headed west down Greenpoint Ave, take a right onto Franklin, then a left onto Kent Street.

      Our facilities are equipped with elevators, as well as several gender-neutral restrooms. ASL (American Sign Language) interpretation will be provided for this event. If you have any accessibility requests, please contact renee@kickstarter.com.

      2 comments
    • Functional Swift Conference

      Swift has given iOS developers a glimpse into a world that we may otherwise have never seen: the world of functional programming. Eighty developers took the time on a cold December day to discuss the technical and philosophical aspects of functional programming, and how we can incorporate these ideas into our everyday work. The conference was hosted in our beautiful theater, with six featured speakers giving 30-minute talks on everything from thinking functionally to using monads to generalize your code.

      If this kind of thing piques your interest, we're hiring for iOS!

      Natasha Murashev — The Functional Way

      Brian Gesiak — Functional Testing

      Justin Spahr-Summers — Enemy of the State

      Andy Matuschak — Functioning as a Functionalist

      John Gallagher — Networking with Monads

      Brandon Williams — Functional Programming in a Playground

      2 comments
    • Pull Requests: How to Get and Give Good Feedback

      We follow a typical GitHub flow: develop in feature branches off master, open pull requests for review and discussion, then merge and deploy. My favorite part of this process is the pull request. I think that the way a team interacts with pull requests says a lot about the healthiness of its communication. Here's what we've learned at Kickstarter about getting and giving good feedback.

      Giving Good Feedback

      Before you begin, consider that someone has potentially spent a lot of time on this code, and you are about to begin a semi-public conversation where you search for problems and suggest ways that the code could be better. You want to be polite, respectful, courteous, and constructive.

      All of these things can be accomplished by being curious.

      Start by searching for places that don't make sense or that act in surprising ways. When you find one, ask a question! Maybe you found a bug, or a way to simplify. In this case, asking allows the submitter to participate in the discovery. On the other hand, maybe the feature has a good reason that you overlooked. In this case, asking gives the submitter an opportunity to explain, and gives you a chance to learn something new.

      Suppose that your question was a misunderstanding, and it's all settled now. Or suppose that you figured out the answer on your own. That's not the end! You can still contribute: think about how the code was different than you expected, and suggest how it might be clarified for the next person.

      The best part of this approach is that anyone can do it at any skill level. You'll either end up contributing constructively or learning something new. Maybe both! There's no bad outcome.

      Getting Good Feedback

      When preparing a pull request, remember that this is not an adversarial situation, it's not a step that you outgrow, and it's not a hurdle on the way to production. Imagine instead that you are a writer submitting a first draft to a panel of editors. No matter your skill level, you will benefit from fresh eyes.

      Before submitting, try to remove distractions. Skim through what you're about to submit. Does it have any obvious stylistic issues, or code left over from previous iterations? Did you make stylistic updates to unrelated code that could be merged and deployed separately? Now's your chance to clean up your branch and let people focus on the good stuff!

      Then write up a good description. Your goal in the description is to describe why you've done this work and how you've approached the solution. But that's not all: this is an opportunity to talk about where you'd like extra eyes and @-mention who you'd like to take a look.

      Once you've submitted and you're waiting for feedback, take yet another look through your code. In some cases you might seed the conversation by commenting on spots where you'd like to draw attention. But don't explain too much just yet! You don't want to spoil those valuable first impressions.

      By now coworkers are engaging with your pull request and beginning to ask questions. Be welcoming! Every question provides an opportunity to learn where your code caused someone to stumble. Remember that if you have to clarify in the pull request, you should additionally consider clarifying the code as well. Your goal is to commit code that not only accomplishes what it intends, but can be understood and maintained by your entire team.

      Teamwork

      When pull requests have healthy communication, they're moments that engineers can relish and look forward to. They provide an opportunity for participants to learn from each other and collaborate on making something of a higher quality than any single person might have achieved on their own. If you feel that pull requests have been missing something in your organization, try out some of these ideas!

    • Engineering Year in Review 2014

      Every year Kickstarter does a Year in Review post to look at what happened over the year. We highlight exciting projects and milestones, and show you the data from that year. This year the Kickstarter Engineering team has decided to do a mini-Year in Review. We think we had a pretty exciting year: we grew, we shipped awesome product (with a lot more to come in 2015), and we had a lot of fun doing it.

      So what did 2014 look like, by the numbers?

      • We started the year with 14 engineers and finished it with 20 engineers (including saying goodbye to three awesome folks).
      • We merged 725 pull requests on the Kickstarter platform.
      • We merged 45 pull requests on our payments processing application.
      • We deployed the Kickstarter platform 9,460 times.

      We also deployed some amazing features, changed the way we worked, and did a lot of other cool stuff. Here's a (small) sample:

      • We rolled out a major update to the look of the Project Page.
      • We hosted Engineering Meetups and the Functional Swift Conference.
      • We created Engineering leads, cross-functional teams, and hired a VP of Engineering.
      • We moved off Amazon and onto Stripe for our payments.
      • We started regular Engineering Hack Days and held another company-wide Hack Day.
      • We added analytics to the emails we send folks.
      • We partnered with Flatiron School to provide resources to students without college diplomas.
      • We created a dedicated Data team.
      • We made it easier for you to discover projects on Kickstarter.
      • We rolled out to a bunch of new countries (including introducing the Fastly CDN to improve international performance and mitigate DDOS attacks).
      • On the security front, we introduced two-factor authentication and converted to 100% SSL to keep y'all safer when using Kickstarter.
      • We extended our GitHub for Poets program to more of the company and had one of our Community team reach 100 commits!
      • We open-sourced our Lunch Roulette algorithm.
      • The White House used some of our code for their Year in Review!

      We've got lots more planned for 2015 and we look forward to sharing it with you! Happy New Year!

      4 comments
    • Growth of Kickstarter's Team and Tech

      I talked recently with Steve from RubyNow about how Kickstarter's Engineering efforts have grown since the first commit in September 2008. We discussed planning for the unknown, managing technical debt, transitioning from small to medium team sizes, and why a (mostly) monolithic Rails app has worked well for us so far. Give it a listen at the RubyNow blog!

    • E-Mail Analytics at Kickstarter

      I spent the summer working as an intern at Kickstarter through the HackNY program in New York City. When I started, Kickstarter was gathering minimal data about e-mail activity on its platform. E-mail is a key part of the lifecycle of a Kickstarter project — it's a crucial part of project updates and backer surveys, for example. 

      My project for the summer was to create a system that Kickstarter engineers could use to gather more granular e-mail data. The goal was to better understand how users are interacting with the e-mails Kickstarter sends.

      Kickstarter uses a cloud-based SMTP provider called SendGrid to manage the delivery of e-mails to users. I built an e-mail analytics system for Kickstarter that integrates with SendGrid's webhooks. Webhooks enable SendGrid to send us data about e-mail events from their service, such as click, open or delivery events. That data is received by Kickstarter, where it is parsed and sent to our data stores. 

      For example, if the recipient of a backer reward survey opens that e-mail, SendGrid is notified of that event. SendGrid then posts data about that event to our callback URL. When Kickstarter receives this data, our system adds additional properties to the data and then sends it along to the analytics service Mixpanel and our data store at Redshift for our further analysis.

      This is one of the first graphs I made using our new e-mail data, which shows the volume of Kickstarter e-mail events by type.
      This is one of the first graphs I made using our new e-mail data, which shows the volume of Kickstarter e-mail events by type.

      We wanted more information about the processed e-mail messages than was offered by SendGrid's event notifications alone. So, we decided to add an X-SMTPAPI header to our e-mails. This header contains data about the associated e-mail, such as a project's id number. SendGrid saves any data encoded into this header, for the purpose of sending it to us as a part of the regular event notification data.

      The data that is sent in the X-SMTPAPI header consists of what SendGrid calls Unique Arguments and Categories. The Unique Arguments can be any data that consists of key-value pairs. The category is a string we send containing the name of the type of email being sent.

      So, we can send Unique Arguments associated with a given email that look like this:


      And we can send the Category for the email like this:

      You might be wondering when we're actually building the X-SMTPAPI header that gives SendGrid all of this useful data. To do that, I built an interceptor class. An interceptor is a feature of Rails ActionMailer, which gives you models and views to use for sending e-mail. An interceptor gives you a way to perform actions on e-mail messages before they are handed off to delivery agents. The interceptor class that I built creates and populates an X-SMTPAPI header for any e-mail marked as needing it. The interface for that class looks like this:


      Once this header is built, the interceptor sends the modified mail message on to the delivery agent as normal. 

      Adding an e-mail to this analytics process amounts to one method call in its mailer method. The method tags an e-mail as needing analytics. The interceptor then adds the X-SMTPAPI header containing our unique arguments and category data to any e-mail with the analytics tag, before finally sending it to SendGrid for delivery to the recipient. 

      So, to sum everything up, this is how the e-mail analytics system I started building this summer works:

      • An e-mail is marked as requiring analytics in its mailer method. 
      • The e-mail is intercepted before it is handed to delivery agents, using a special interceptor class that builds the X-SMTPAPI header. 
      • SendGrid sends the e-mail.
      • SendGrid receives event notifications about the e-mail (delivery, click and open), which are then posted to our callback URL by their webhooks. 
      • Our callback URL parses the event notification data and hydrates it with new, richer data. 
      • The hydrated e-mail event data is sent to Mixpanel and Redshift for further analysis.

      I came into this project with very little knowledge of the complexities of e-mail delivery and tracking. This new feature touches many parts of Kickstarter's larger e-mail system, so having the chance to build something that could interface with that in the simplest way possible was a fun challenge. It's also exciting to have built a resource that enables Kickstarter to understand what kinds of e-mails are most meaningful to users — by analyzing open rates, for example. I'm happy that I succeeded in seeing this feature through to completion!

      6 comments
    • Building our new engineering job ad

      At Kickstarter we try to be thoughtful about culture and hiring. As a recent addition to the team I took up the task of drafting a new job ad for engineers. We decided our new ad needed to satisfy three criteria:

      • It should reflect our culture and values.
      • It should clearly explain the job we're offering, the person we're seeking and the environment you can expect at Kickstarter.
      • It should make the hiring process clear and transparent.

      One of my colleagues had previously suggested this job ad from the team at Hacker School as an example of an exceptional job posting that had really resonated with him. 

      I decided to adapt the Hacker School posting including reusing much of their walkthrough of the interview process. Our interview process is identical but their explanation of the process is elegant, simple and clearly sets expectations.

      *UPDATE* - When we posted the job ad and this blog post we didn't do an awesome job of acknowledging our reuse of some of Hacker School's job posting directly. The mistake was totally mine and I want to sincerely apologize for that omission. We've updated this blog post and the job posting to acknowledge that debt and linked to the original content.

      Our job ads are part of the Rails application that powers the Kickstarter platform so they are added to the site via a Git Hub pull request. I wrote the draft ad and created a pull request for it.

      Pretty much immediately the team began to review it much like they review any other code on our site. Like code reviews, the responses covered style, content and implementation (I managed not to mess up any code which was excellent as it was also my first commit at Kickstarter). And the review wasn't just limited to the Engineering team, our Director of HR also reviewed the pull request.

      In the review, the team immediately hit on the areas which we feel most strongly about like diversity.

       And inclusivity and accessibility.

      As well as ensuring we make clear the type of engineers and skills we're seeking.

      Throughout the process we iterated (and rebased) the pull request as people provided feedback and input. As a team we're pretty proud of the final product. We think both that the content and the process provide an accurate reflection of what makes Kickstarter Engineering successful and an awesome place to work. 

      1 comment
    • Refactoring for performance

      Frequent Kickstarter visitors may notice that pages load a bit faster than before. I'd like to share how we refactored an expensive MySQL query to Redis and cut over 100ms off typical load times for our most prolific backers.

      If you're logged in, there's probably a little green dot next to your avatar in the top right of the page. That's your activity indicator, letting you know there's news about projects you've backed. There's a similar interface in our iPhone app that displays the number of unseen activity items.

      We implemented that feature years ago in the most straightforward way. We have a MySQL table that joins users to each activity item in their feed with a boolean field to indicate if the user has seen the item.

      That worked fine when our site was smaller. But as Kickstarter grew to millions of users, some with thousands of activity items, the query performance degraded. Performance was especially poor for our most prolific backers.

      We've had prior successes using Redis for quick access to denormalized data, so it seemed like a natural solution.

      At first, we weren't sure what Redis data structures to use to store all the counters. Should we use a simple string key per user, or one big hash for all users? With a little research about the hash-max-ziplist-entries config and some benchmarking, we realized that putting counters in hashes in groups of 512 yielded great memory usage with negligible CPU impact.

      Here's the annotated code my colleague Lance Ivy and I implemented. It's an example of the clear, maintainable, testable Ruby code we like to write.

      Here's the interface:


       indicator = Indicator::UserActivity.new(user_id)
      
      # When a new activity is added to a user's feed: 
      indicator.increment
      
      # When a user views their unseen activity: 
      indicator.clear
      
      # Does the user have any unseen activity? 
      indicator.set?
      
      # How many unseen activities does the user have? (for the iOS app)
      indicator.value

      Deploying

      While the code change was small, the indicator is rendered on virtually every page. Here's how I rolled out Redis indicators in small, easily verifiable parts:

      • Start writing new activity to Redis, while reading from MySQL
      • Backfill activity counts for Kickstarter staff from MySQL to Redis
      • Enable only staff users to read from Redis
      • Backfill all user counts to Redis (took ~2 days)
      • Enable reading from Redis for 10/50/100% of users in 3 quick deploys
      • Play Killer Queen

      Graphs

      So did this refactor make the site faster? Most definitely. Let's look at some graphs.

      I'll focus on the homepage where returning users often land. Here's average homepage render time before and after the refactor:

      Mean homepage performance for all users. Click to embiggen.
      Mean homepage performance for all users. Click to embiggen.

      We shaved ~60ms off the average load time by eliminating the Activity#find call.

      Note that the three deploy lines between 11:40 AM and 12:10 PM are when we enabled Redis indicators for 10/50/100% of users.

      The NewRelic graph is a mean of both logged in and logged out users (who of course don't have any activity). It's not particularly indicative of a user's experience. Here's a more detailed graph of homepage performance for just logged in users.

      Homepage performance for logged in users. Click to embiggen.
      Homepage performance for logged in users. Click to embiggen.

      Not only did we improve performance, but we greatly reduced the variance of load time. The site is faster for everyone, especially prolific backers.

      5 comments
    • Followup: Kickstarter Engineering Meetup

      photo by @chenjoyv
      photo by @chenjoyv

      Last Thursday we held a meetup at our office to discuss maintaining positive, inclusive engineering cultures, and it went swimmingly! Our team really enjoyed meeting people from other companies in the area and exchanging ideas about work environments, tools that we use, and everything in between. 

      Here's a brief summary of the talks given with links to their slides:

      Thanks to everyone who attended. Hope we see you again!

      Leave a comment
    • Kickstarter Engineering Meetup - Aug 21

      We’re hosting an engineering meetup at our HQ in Greenpoint! We want to discuss how to create and sustain positive, inclusive engineering cultures. We believe the conversation should include a diverse range of perspectives.

      Who: All are welcome to attend! We have three speakers lined up, each giving a 10-15 minute presentation:

      What: We’ll talk about the processes and tools that enable positive engineering cultures. Following the presentations, Kickstarter staff will facilitate loosely organized break-out discussions to make it easy for everyone to participate. We’ll provide snacks, beer and non-alcoholic beverages.

      When: Thursday, August 21, from 7 to 9pm

      Where: Kickstarter, 58 Kent St, Brooklyn. You can take the East River Ferry to Greenpoint or take the G train to Nassau Ave. and then walk or take the MTA shuttle bus to Greenpoint.

      How: Please RSVP here.

      Accessibility: The Kickstarter office is wheelchair-accessible. A professional transcriptionist will transcribe the presentations in realtime. The Kickstarter office has three non-gendered single-occupancy bathrooms.

      Kickstarter is committed to ensuring a safe, harassment-free environment for everyone. Please contact us if you have any questions or concerns.

      3 comments
    • Lunch Roulette

      Kickstarter's culture is a core part of who we are as a company and team. Our team hails from a hugely diverse set of backgrounds – Perry was working as a waiter at Diner when he met Yancey, most of our engineers studied liberal arts (myself included – Philosophy), and our community team is made up of former and current projectionists, radio hosts, teachers, funeral directors, chefs, photographers, dungeon masters, artists, musicians, and hardware hackers. Last year, we had the idea to facilitate monthly lunch groups as a way to see if we could accelerate the kind of inter-team mixing that tends to happen in the hallways and between our normal day to day work.

      In addition, groups would be encouraged to go for a walk, find a new place in the neighborhood to have lunch, and Kickstarter would pick up the tab.

      Shannon, our office manager at the time, and now our Director of HR, had the unenviable job of coming up with all of these lunch groups. The idea was to make them pseudo-random, so that staff wouldn't end up having lunch with the person they sat next to every day, and that, ideally, they'd meet people they'd never normally interact with as part of their day to day responsibilities.

      And, as our headcount has grown – we've hired half of Kickstarter between February 2013 and now – we also hoped that these lunches could introduce new staff to old.

      But Shannon quickly discovered that creating multiple sets of semi-random yet highly-varied lunch groups was not a trivial task!

      One of the biggest issues with keeping groups interesting was moving a person from one group to another meant a cascade of changes which were tedious, and sometimes impossible to reconcile by hand.

      So, after spending an entire weekend churning out six possible sets of a dozen groups of 4 people each, Shannon took me up on my offer to help build a formal algorithm to help automate what we had been calling Lunch Roulette.

      We put together a meeting and sketched out some constraints that a minimally viable Lunch Roulette generator would have to satisfy:

      • Lunch groups should be maximally varied – ideally everyone in a group should be from a different team
      • Groups should avoid repeating past lunches
      • We should be able to define what it means for a group to be varied
      • It should output to CSV files and Google Docs

      After a couple weeks of hacking together an algorithm in my spare time, I arrived at something that actually worked pretty well – it'd take a CSV of staffers and spit out what it thought were a set of lunch groups that satisfied our conditions.

      We've been using it for over 6 months to suggest hundreds of lunch groups and have been pretty happy with the results, and today I'm open sourcing it. But first, a little more about the algorithm.

      The Fun Part is How Lunch Roulette Works

      Lunch Roulette creates a set of lunches containing all staff, where each group is maximally varied given the staff's specialty, their department, and their seniority.

      It does this thousands of times, and then ranks sets by their overall variety. Finally, the set of lunch groups with highest total variety wins.

      Command Line App

      Lunch Roulette is a command line application that always requires a CSV file with staff "features", such as their team and specialty and start date. It is run using the ruby executable and specifying the staff via a CSV file:

          ruby lib/lunch_roulette.rb data/staff.csv
      

      Features are things like the team that a staffer is on, or the day they started. These features can be weighted in different ways and mapped so that some values are "closer" to others.

      Along with specifying the various weights and mappings Lunch Roulette users, configurable options include the number of people per group, the number of iterations to perform, and the number of groups to output:

          Usage: ruby lunch_roulette_generator.rb staff.csv [OPTIONS]
              -n, –min-group-size N           Minimum Lunch Group Size (default 4)
              -i, –iterations I               Number of Iterations (default 1,000)
              -m, –most-varied-sets M         Number of most varied sets to generate (default 1)
              -l, –least-varied-sets L        Number of least varied sets to generate (default 0)
              -v, –verbose                    Verbose output
              -d, –dont-write                 Don't write to files
              -h, –help                       Print this help
      

      A Dummy Staff

      So that you can run Lunch Roulette out of the box, I've provided a dummy staff (thanks to Namey for the hilariously fake names) dataset in data/staff.csv:

        user_id,name,email,start_date,table,team,specialty,previous_lunches
        4,Andera Levenson,andera@cyberdyne.systems,10/12/2011,3,Operations,,"1,10"
        48,Brittani Baccus,brittani@cyberdyne.systems,12/16/2013,3,Product Manager,,"6,11"
        59,Campbell Russell,campbell@cyberdyne.systems,11/25/2010,2,Community,,"1,5"
        35,Carolina Credo,carolina@cyberdyne.systems,6/6/2010,2,Communications,,"12,1"
        36,Colin Rigoni,colin@cyberdyne.systems,11/18/2013,4,Community,,"12,6"
        44,Collen Molton,collen@cyberdyne.systems,12/17/2013,2,Executive,,"0,10,1"
        12,Cornelius Samrov,cornelius@cyberdyne.systems,9/5/2011,2,Product Manager,Backend,"12,4"
        21,Damion Gibala,damion@cyberdyne.systems,2/16/2013,2,Engineering,,"2,13"
        60,David Graham,david@cyberdyne.systems,12/3/2013,4,Product Manager,Backend,"3,6"
      

      The use of a CSV input as opposed to a database is to facilitate easy editing in a spreedsheet application (a shared Google Doc is recommended) without the need for migrations or further application bloat. This allows non-engineers to add new staff, collaborate, and add new columns if needed.

      Accordingly, the date format of MM/DD/YYYY is specific to common spreadsheet programs like Google Docs and Excel.

      Note: There's a difference between team and specialty. While you and another person might have the same specialty, you might be on different teams. By default Lunch Roulette puts precedence on preventing two people with the same specialty from having lunch, since you probably work closer together than people on the same team. The previous_lunches column contains a double quoted comma-delimited list of previous lunches each having their own ID. If no previous lunches have taken place, then ids will be generated automatically (see the CSV Output section below for more info). All users need to have a user_id to help Lunch Roulette, but this can be an arbitrary value for now.

      Configuring Lunch Roulete

      Mappings

      At the minimum, Lunch Roulette needs to know how different individual features are from each other. This is achieved by hardcoding a one dimensional mapping in config/mappings_and_weights.yml:

        team_mappings:
          Community Support: 100
          Community: 90
          Marketing: 80
          Communications: 70
          Operations: 50
          Product: 40
          Design: 30
          Engineering: 20
          Data: 0
        specialty_mappings:
          Backend: 0
          Data: 20
          Frontend: 30
          Mobile: 50
          Finance: 100
          Legal: 120
        weights:
          table: 0.6
          days_here: 0.2
          team: 0.9
          specialty: 0.1
        min_lunch_group_size: 4
      

      Lunch Roulette expects all employees to have a team (Community, Design, etc.), and some employees to have a specialty (Data, Legal), etc.

      Caveat Mapper

      These mappings are meant to provide a 1-dimensional distance metric between teams and specialities. Unfortunately, the results can come out a little arbitrary – e.g., why is Community so "far" away from Engineering? I have some notes below about how I might fix this in future versions, but this approach seems to work well enough for now given the intent of Lunch Roulette. Having put a lot of thought into the best strategy for quantizing how teams and colleagues may differ, I'll say that almost all solutions feel unpalatable if you think about them too hard.

      Weights

      You should also specify the weights of each feature as a real value. This allows Lunch Roulette to weight some features as being more important than others when calculating lunch group variety. In the supplied configuration, team is weighted as 0.9, and is therefore the most important factor in determining whether a lunch group is sufficiently interesting.

        weights:
          table: 0.6
          days_here: 0.2
          team: 0.9
          specialty: 0.1
      

      It's not strictly necessary to keep the weights between 0 and 1, but doing so can keep scores more comprehensible. Finally you can specify the default minimum number of people per lunch group:

        min_lunch_group_size: 4
      

      When the number of total staff is not wholly divisible by this number, Lunch Roulette randomly assigns remaining staff to groups. For example, if a staff was comprised of 21 people, and the mimimum group size was 4, Lunch Roulette would create four groups of four people, and one group of five people.

      Determining Mappings

      The weights that Lunch Roulette uses to calculate team variety are specified in the config/weights_and_mappings.yml file. Team and specialty mappings effectively work as quantizers in the Person class, and if you add new features, you'll have to modify it accordingly.

      For example, Community may be culturally "closer" to the Communications team than the Engineering team. I highly recommended that you tweak the above mappings to your individual use. Remember, the more you define similarity between teams and specialties the easier it is for Lunch Roulette to mix people into varied lunch groups. Seniority is calculated by subtracting the day the employee started from today, so staff that start earliest have the highest seniority. Not all staff are required to have values for all features. In particular, if each staff has a specialty, Lunch Roulette may have a difficult time creating valid lunch sets, so it's recommended that no more than 30-40% have specialties.

      Previous Lunches and Validations

      Before Lunch Roulette calculates a lunch group's variety, the LunchSetclass attempts to create a set of lunches that pass the validations specified in the class method valid_set. For a staff of 48 people with a minimum group size of 4, a set would contain a dozen group lunches. Out of the box, there are three validations Lunch Roulette requires for a set to be considered valid:

      • The set cannot contain any group where 3 or more people have had lunch before
      • The set cannot contain more than one executive (a dummy previous lunch with the id of 0 is used here)
      • The set cannot contain anyone with the same specialty (remember, specialties are different than teams)

      In most scenarios with at least one or two previous lunches, it is impossible to create a valid lunch set without at least one group having one pair of people who have had lunch before.

      Choosing a Set

      Remember, the set with the most heterogeneous lunch groups win. variety is first calculated within groups, and then across sets. The set with the highest variety wins.

      Group variety

      Once a valid lunch set is created Lunch Roulette determines the variety of each group within the set thusly:

      1. Choose a feature (e.g. we will try to mix a lunch group based on which teams people come from)
      2. Choose a person
      3. Normalize the value of that person's quantized feature value against the maximum of the entire staff
      4. Do this for all people in the current group
      5. Find the standard deviation of these values
      6. Multiply this value by the configured weight
      7. Repeat this process for all features
      8. The group score is the sum of these numbers

      The resulting average is a representation of the how different each member of a given group is from each other across all features and can be seen in the verbose output:

        Tony Reuteler (Design, Table 2),
        Campbell Russell (Community, Table 2),
        Idella Siem (Product Manager, Table 2),
        Fred Pickrell (Community, Table 3)
        Emails:
          tony@cyberdyne.systems,
          campbell@cyberdyne.systems,
          idella@cyberdyne.systems,
          fred@cyberdyne.systems
        Sum Score: 0.4069
        Score Breakdown: {
          "table"=>0.075,
          "days_here"=>0.04791404160231572,
          "team"=>0.28394541729001366,
          "specialty"=>0.0
        }
      

      The higher the sum, the more varied that particular lunch group.

      Set variety

      Since all sets will have the same number of groups in them, we can simply sum the average scores across all the groups and generate at a per-set score. This represents the average variety across all groups within a set and is used to compare sets to each other.

      Formally

      I was interested in how Lunch Roulette could be represented formally using math, so I asked my colleague Brandon – Kickstarter's math-PhD-refugee-iOS-dev – for some help. After some beers and a whiteboard session, we arrived at at decent generalization of what's happening under the hood. It should be noted that any errors in the following maths are entirely my fault and any brilliance should be entirely ascribed to Brandon's patient insights.

      Let [inline_math]S[/inline_math] be the set of staff and [inline_math]\mathscr{U}[/inline_math] be the set of all partitions of [inline_math]S[/inline_math] into [inline_math]N[/inline_math] groups. Since [inline_math]\mathscr U[/inline_math] is very large, we narrow it down by throwing out lunches that we consider boring. For example, no lunch groups with 3 or more people who have had lunch before, etc.

      Then we are left with [inline_math]\mathscr U'[/inline_math], which is the subset of [inline_math]\mathscr U[/inline_math] of valid lunches.

      We define a "feature" of the staff to be a integer-valued function on [inline_math]S[/inline_math], i.e. [inline_math]f : S \rightarrow \mathbb Z[/inline_math].

      For example, the feature that differentiates teams from each other might assign 15 to someone on the Community, and 30 to someone on the operations team.

      It's important to note that this number doesn't represent anything intrinsic about the team: it's merely an attempt at mapping distance (albeit one-dimensionally) between teams. Future versions of Lunch Roulette should probably switch this to a function returning a vector of values encoding multi-dimensional characteristics about teams (e.g. [inline_math][0,1,0,1,1,1,0][/inline_math]).

      Let's fix [inline_math]M[/inline_math] such features, [inline_math]f_i : S \rightarrow \mathbb Z[/inline_math]. For a given feature [inline_math]f[/inline_math], let us define: [inline_math]||f|| = \max\limits_{s \in S} f(s)[/inline_math]

      We need [inline_math]||f||[/inline_math] so that we can normalize a given feature's value against the maximum value from the staff.

      It is also useful to apply weights to features so that we can control which features are more important. Let [inline_math]W_i \in [0,1][/inline_math] be the set of weights for each feature [inline_math]i=1, \ldots, M[/inline_math].

      Then we maximize the variety among lunch groups thusly: [math] \max\limits_{\mathscr{G}\in\mathscr{U}'}\sum\limits_{G\in\mathscr{G}}\sum\limits_{i=1}^{M}\sigma\left(\dfrac{f_i(G)}{||f_i||}\right)\cdot W_i [/math] In english, that's: for each feature inside each group, we normalize a person's value to the maximum value found in the staff, then calculate the standard deviation ([inline_math]\sigma[/inline_math]).

      Then, all [inline_math]\sigma[/inline_math] are added up for the group and then multiplied by the weight given to the feature to achieve an overall variety metric for a given group. The sum of those [inline_math]\sigma[/inline_math]s inside a set represents its overall variety.

      Here's the inner loop (representing the innermost [inline_math]\sum[/inline_math] above) in LunchGroup which calculates a given group's score:
          def calculate_group_score
          h = features.map do |feature|
            s = @people.map do |person|
              person.features[feature] / config.maxes[feature].to_f
            end.standard_deviation
            [feature, s * config.weights[feature]]
          end
          @scores = Hash[*h.flatten]
        end
        
      Lunch Roulette does this thousands of times, then plucks the set with the highest overall score (hence the [inline_math]\max_{}[/inline_math]) and saves that group to a CSV.

      I've since discovered that lunch roulette is a version of a "Maxmimally Diverse Grouping Problem", and it seems like some researchers from The University of Valencia in Spain have built some similar software to Lunch Roulette in Java using a couple different methods for maximizing diversity.

      Gut Testing Lunch Roulette

      If specified, Lunch Roulette will output the top N results and/or the bottom N results. This is useful for testing its efficacy: if the bottom sets don't seem as great as the top sets, then you know its working! This will output 2 maximally varied sets, and two minimally varied sets:

        ruby lib/lunch_roulette.rb -v -m 2 -l 2 data/staff.csv
      

      If you wanted to get fancy, you could set up a double blind test of these results.

      CSV Output

      Unless instructed not to, Lunch Roulette will generate a new CSV in data/output each time it is run. The filenames are unique and based off MD5 hashes of the people in each group of the set. Lunch Roulette will also output a new staff CSV (prefixed staff_ in data/output) complete with new lunch IDs per-staff so that the next time it is run, it will avoid generating similar lunch groups. It is recommended that you overwrite data/staff.csv with whatever version you end up going with. If used with the verbose option, Lunch Roulette will dump a TSV list of staff with their new lunches so you can paste that back into Google Docs (pasting CSVs with commas doesn't seem to work).

      Take It For a Spin

      I've open sourced Lunch Roulette and it's available on GitHub under a MIT license.

      Thanks

      Lunch Roulette went from a not-entirely serious side project into something much more interesting and now, I hope, something possibly useful for others. But I couldn't have done it without Shannon Ferguson having done all of the work manually, and Brandon Williams helping me with the math.

      Please consider forking it and letting us know if you use it.

      Happy Lunching!

      6 comments
    • Data @ Kickstarter: A Guided Tour

      A couple of weeks ago, Jeremy and I invited the NYC data community to hear about how the data team works at Kickstarter:


      We’ll focus on the technical architecture behind Kickstarter’s data pipeline and how we use a variety of data sources (from messy CSVs to the results of 100-line Redshift queries running over billions of rows of data) combined with some good-ole-fashioned sleuthing to answer our team’s hardest questions.


      Then we’ll present our internal workflow, how we manage data requests with Trello, and how we document and share our results with the rest of our company using tools like GitHub, ggplot, and knitr.


      We’ll also go over some of the common (and uncommon) questions we try to answer every day, and demo some of our internal products that help us do our day-to-day research.


      We had such a positive response from the people who attended -- so many people asked for the slides -- that we decided to post them below:

      If this is the kind of slide deck that gets you excited, then don't miss our two open positions for a data analyst and a data engineer!

      1 comment
    • Coordinating with Cards

      The most effective tools are the ones that enhance the way you already work. Of course, no tool will be perfect, but some will fit your process better than others. I’m going to share a bit about how Kickstarter works, and how we use Trello to support our style. 

      How We Work

      While there are a lot of cool things about the Kickstarter Product & Engineering team, two that stand out to me are the openness and the flexibility. 

      Openness

      Kickstarter as a whole tries to be open about our work. At the beginning of the year, Yancey shared our company priorities. Each week at our All Hands meeting, a few individuals stand up to talk about a project that they’re working on.

      This translates into our engineering work. We have a status thread each week so everyone can share what they’re working on. We open early pull requests to get lots of feedback on projects. Our deploys and every commit to master automatically post to a shared Campfire room, so everyone can see what’s changing. Whenever we ship something new, we email the entire company.

      We want everyone to be aware of what’s going on, and be able to add input at the right time. 

      Flexibility

      We also try to maintain flexibility. We haven’t split our team into smaller groups for specific areas of the product. Instead, for each new project we form an ad-hoc team based on who’s available and what people want to learn. Usually it consists of a PM, a designer, at least one engineer, and someone from the community team.

      This approach works because it enables people to get experience on different pieces of the product. While people might still have an area of expertise, there shouldn’t be anything that only one person knows how to do. It also adds new perspectives to each project.

      Plus, each of these teams tends to find their own way to be most effective. Some meet frequently, some don’t. Some start Campfire rooms, some just gchat each other. Each group finds its own way to get things done. 

      What We Wanted

      Historically, we used a Google Docs spreadsheet to track the overall project status, and Assembla tickets to work items. Unfortunately, those tools weren’t the best support for our processes - they inherently didn’t mirror our flexibility and openness. The spreadsheet didn’t fit into most peoples’ workflow - they’d have to remember to go look for it, and often didn’t. Assembla was great for ticket tracking, but didn’t give us an overall view of what was happening.

      We started looking around for new tools in 2013. The main goals were:

      • Enable feature teams to be flexible, and have a custom workflow. 
      • Allow everyone to see what was going on in other projects.

      Trello was one of the tools we experimented with.

      First, the iOS feature team used Trello to manage their release. Then, Fred Benenson started using Trello to track data requests from other people (both from Product & Community!) We also saw UserVoice’s article about how they were using Trello to plan a roadmap. That meant we could use one tool to replace our spreadsheet and our Assembla tickets!

      Given the successful experiments, we felt it was worth investing more in having our entire team use Trello. We’ve now been doing that for a little over a year. Our current system works pretty well in our process - it gives individual teams flexibility, and let’s everyone see the status of each project. Using Trello

      Using Trello

      For those who are unfamiliar, Trello has a few key concepts: Boards, Lists, and Cards. Each of these concepts is flexible - within each board, you can have any number of lists (named how you wish) and as many cards as you’d like. We’ve adopted ways of using each aspect that fits with how we work.

      I’m going to walk through how we do this from the overall project standpoint. Keep in mind that we also use Trello within our small teams - but each team might have their own lists or levels of granularity for cards.

      Our Project Cards

      The basic unit within Trello is the card. Each card has a title, short description, and the ability to add comments or attach images. You can also add people to the card and use colored labels to tag the card.

      We start with one card for each Project. “Project” is a loose term. It covers everything from “Create Advanced Search” to specific production bugs that we’re tackling.

      The card is the central place to keep track of the project. That doesn't mean everything has to live on the card, it just has to be available from there. If there’s an associated document we’ll store it in Google Drive, but there's always a link from the card. We attach mockups to the project card, but they may also be in a .psd somewhere.

      The benefit of the project card is that anyone within the company can go to it to get an accurate representation of the status. Each person working on a project joins the card, so it’s easy to see who’s working on what, and who to ask questions.

      We organize project cards into a few major boards: Current Projects, Product Backlog, and Engineering Backlog. 

      Current Projects Board

      The Current Projects board is where you can find anything that we’re working on. “Working on” means that we’ve started to invest engineering resources. Our Current Projects has four lists:

      • Priority Projects 
      • Side Projects 
      • Support & Production Bugs 
      • Shipped

      Priority Projects

      Priority projects are the major efforts we’re taking on. These are usually multi-month projects. Shipping a new version of the Start Page fell into that column, as did our Billion Dollar page. We rank the projects in this column by their importance. If you’re on more than one card in the column, the one that’s highest is the one you do first.

      Priority projects are usually more complex than one card can represent. When that’s the case, we open a new “Project Board.” That Project Board will contains cards for granular work items. The project team would spend most of their time in that project specific board, instead of on the Current Projects board.

      The Product Manager for the feature handles updating the project card at least once a week. The card should always have the latest information, so the information will be available to everyone. This way the project card serves as a summary of the status, and the project board serves as the project team’s workspace.

      Side Projects

      Even with two priority projects, there can be downtime for other work. That’s where side projects come in. These are smaller projects that are done between larger things. A recent example was the engineering work that went into adding new subcategories. These often only take a few days.

      Usually, these projects don’t need their own board. They often have a discussion in the comments between PM, design & dev to ask any clarifying questions or document changes.

      Recently, we’ve been taking the time to share many items that might be considered “side projects” during internal lightning talks. This gives others a chance to give feedback on the projects before they are in production.

      Support & Production Bugs

      Support & Production Bugs is exactly what it sounds like. Our support team adds cards to the column when a user writes in with a replicable issue. Having support & production bugs with the rest of the project cards means the support team is always seeing what engineering is up to.

      Each week a different developer is the “Support Dev” and works on resolving those issues. That said: they don’t need to fix each issue themselves, but they either fix it, or find someone who can. These cards usually contain a lot of direct discussion between the Support Dev and the Support team. They tend to be resolved within a week, but sometimes will carry over into the next week.

      Shipped

      Once any project is done - be it a priority, side project, or a bug - we move it into the “Shipped this week” column.

      We bump our shipped lists down, and after two weeks, we archive them. We keep three lists of “shipped” around because it’s difficult to go through archived items in Trello. After a few weeks we don’t need to revisit the cards.

      Product Backlog Board

      We also keep a board for all the work we want to do at some point. This board is our Product backlog.

      We’ve tried organizing this board a bunch of different ways.

      Roadmap

      Right now we’ve moved to a system where the first column is our roadmap. It’s the “next 10 priority projects” (in ranked order).

      The entire PM team and representatives from around the company select the next 10 projects. For a project to get into the “priority” column on the current projects board, it needs to first be in the list of the next 10 projects.

      We also tried organizing the Product Backlog by quarter. We slotted the major projects out by quarter (four lists), instead of just in a stacked list. Both of these solutions have worked well.

      Topics

      After the first column, we organize all other possible projects by topic area. A few examples of those lists are: Creator Resources, Backer Tools, and Project Page.

      The PM for each area maintains their topical lists. Other team members are free to email ideas, add comments to the cards, or even add new cards. This helps us keep track of the ideas & discussions for when we have time to start work on the project.

      Some of the items in the topical lists will move to “next 10 projects.” Smaller items might move directly to side projects as people have time.

      Engineering Backlog Boards

      We keep our Engineering backlog separate from our Product backlog. The Engineering backlog focuses more on infrastructure work that we want to do. It includes things like any work our DevOps team does, and improvements to our payments platform.

      Once we’ve decided to do a piece of Engineering work, we also move it to the Current Projects board.

      Project Boards 

      Of course, this only helps us keep track of projects at the high level. We can’t keep each individual work item in the Current Projects board, or it would be difficult to get a good idea of what’s going on.

      Instead, individuals teams will open their own project boards. For instance, “Advanced Search” had its own board where we could keep track of individual work items. While the individual team will open the board, anyone who is interested in seeing more of the details of the work is free to join the board and get updates.

      Each team opts to use their board a bit differently. One popular arrangement is to sort the work items by status - with columns labeled accordingly: “to do” “in progress” “needs testing” “done” columns. Some teams might opt to split the “to do” items by different functions, or by who needs to get them done.

      Each of these project boards also last for different lengths of time. “International” is a longer living board as we open Kickstarter to new countries. On the other hand, some boards like “Advanced Search” can be closed when the project ends.

      Using Trello gives teams the flexibility to decide what structure they’d like to have in their board, and follow that. Overall, Trello has helped us support our process - it lets us be open about what we’re working on, without overly constraining how teams work.

      2 comments
    • GitHub for Poets


      po·et ˈpōət,ˈpōit noun 1. You're not a programmer (yet). But you want to have a hand in crafting the code that runs our site. That's awesome and you're awesome.


      Recently, our operations engineer Aaron Suggs had an amazing idea that would peel back the curtain a bit from the engineering team workflow. What if we gave a workshop to interested non-dev Kickstarter employees (“poets”) on how to engage with our code via GitHub's web flow? This GitHub tool eases the burden of performing the various Terminal.app commands that can be intimidating to beginners. Teaching employees how to perform code changes this way could have several benefits including but not limited to:

      • Providing the tools for interested employees to make copy edits 
      • Empowering members of Kickstarter's Community and Operations teams to participate in the world of code 
      • Eliminating developer unavailability as an obstacle to making safe changes

      I co-taught the course because my own entrance into the world of programming had similar ideological roots. I originally worked on the Community team at Kickstarter, but after months of enthusiastic code consumption and a few features deployed to production, I moved to our engineering team full-time. An accessible, inviting, and compassionate engineering team culture can change lives!

      After putting together a rough syllabus, Aaron and I held 4 separate “GitHub for Poets” sessions attended by 5-8 different employees from various teams in the company. The sessions had individual flavors; one was fueled by a need to make a copy edit on our Support navigation bar that was eventually merged. Another was led by the desire to make silly changes to the homepage for ultimate "Wow!" factor, which I personally supported because the changes were so visible and exciting (though they remained unmerged):

      In all cases, the structure of each hour-long course unfolded as follows: 

      1. Healthy programming attitudes (scratch your own itch, don't-repeat-yourself, etc)
      2. Introduction to our tech stack (HAML, Sass, JavaScript, Ruby)
      3. Rails directory structure
      4. Creating feature branches via the GitHub UI
      5. Finding files, editing, and committing changes with helpful commit messages
      6. Opening a pull request for the feature branch

      We taught each of these sections to the enthusiastic maximum, and each employee was encouraged to add commits to the branch. The mock pull requests went out to the whole dev team, who responded enthusiastically with their comments, suggestions, and emoji. Good vibes abounded!

        Since the first round of GitHub for Poets courses ended, multiple employees who aren't on the engineering team have made commits that were ultimately merged, including changes to our jobs page, policy pages, and support resources. One of these changes even touched some Ruby code. We require each potential change to be done within a feature branch, pull requested, and merged by a developer themselves, but these little bits of process are no hindrances to the passion of poets.

        Working for a tech company involves relying on code that fuels our jobs, our users, and our community, but it often happens that only 20% of the company can touch that same code! By encouraging personal responsibility, a willingness to ask for help, and constructive feedback between engineers and poets, any startup can help open the doors to more inclusive development.

      7 comments
    • Hierarchy of DevOps needs

      Operations teams have many responsibilities: monitoring, provisioning, building internal tools, and so on.

      To better understand these responsibilities, I organize them as a hierarchy of needs.

      Hierarchy of DevOps needs
      Hierarchy of DevOps needs

      Low levels of the pyramid are basic needs, and prerequisites for higher levels. Higher levels are more personally fulfilling and valuable to your organization.

      The goal is to deal with low level issues effectively so you can spend most of your attention at higher levels.

      For example, if you’re one server crash away from losing your data (backups), it hardly matters if you make deployments faster (team efficacy). Fixing your backups is the priority.

      At each level of the pyramid, there’s lots of work to do and successful careers to be had. As your organization evolves, you'll inevitably need to attend to each tier. A high-functioning team solves issues thoroughly and effectively, so they can focus their attention up the pyramid.

      Even if your job is focused on a single tier, say, security, there are ways to move your attention up the pyramid. You can use frameworks, static analysis, and linting tools to prevent many types of security vulnerabilities (team efficacy). You can use blameless post mortems for training and remediation (team happiness).

      This hierarchy was inspired by Maslow’s hierarchy of needs about human motivation, and Heinemeier Hansson’s levels of aspiration for programmers.

      It's a useful model for ranking work to be done, and gauging the effectiveness your operations team.

      Leave a comment
    • Reporting iOS crashes to StatsD

      Here at Kickstarter we use Crashlytics for our crash reporting in the iOS app. It serves us well and does a lot of work that we don’t want to have to worry about. However, it also introduces more fragmentation into our toolset for monitoring our services. We already have an extensive set of StatsD metrics that we monitor, so it would be nice to see graphs of crashes right next to graphs of our HTTP 200s and 500s.

      Crashlytics provides a delegate method -crashlytics:didDetectCrashDuringPreviousExecution: that is called when the library has detected that the last session of the app ended in a crash. We use this method to hit an endpoint on our servers that will increment a StatsD bucket. The request contains info about the build number and iOS version at the time of the crash, and we include that in the StatsD bucket:


      STATSD.increment "ios.crash.#{build}.#{ios_version}"
      

      Now we can build a graph that shows the total number of crashes over time, or break out into many graphs for crashes per iOS version or app version. The graphs help us to tell a story about the overall stability of our releases, or what fixes have been effective.

      For example, when looking at the total number of crashes over time, we can clearly see an increase in mid September:

      Total # of crashes
      Total # of crashes

      This is right around the time iOS 7 came out. In order to confirm this we can look at a graph that breaks out crashes per iOS version:

      Crashes by iOS Version
      Crashes by iOS Version

      Now it is very clear that iOS 7 has elevated crash rates, though each minor release has been slightly better. For example, the first release of iOS 7 seemed to have a bug in CFNetwork causing hundreds of crashes a day in a function named mmapFileDeallocate. This crash has not happened since 7.0.2, which is reflected in the red and purple curves.

      Since the crash rates are still higher than what we saw in iOS 6 we looked for other ways to workaround the most common crashes. One of the more perplexing iOS 7 specific crashes we see has to do with data detectors in UITextView instances. It happens seemingly randomly, and has occurred in every version of iOS 7. In our most recent update we wrote a UITextView subclass that provided a custom implementation of data detectors in hopes of getting around this crash. The benefits of this work can now be seen by looking at a graph of crashes by build #:

      Crash by build #
      Crash by build #

      Build #510 (the blue curve) is the first build with the fix, and it has the lowest number of crashes we’ve seen for awhile.

      This form of crash tracking has been very useful to us. In fact, it’s become so important that we put a version of these graphs on our deploy dashboard so that we can immediately see if an API or mobile web change affects the crash rate of the app. By leveraging the tools and infrastructure that we are already comfortable with from our web app we allow every engineer to take part in the monitoring and diagnosing of iOS app problems.

      3 comments
    • Drew Conway on Large-Scale Non-Expert Text Coding

      As part of our series of informal engineering and data focused talks at Kickstarter, we hosted Drew Conway to present on his PhD thesis large-scale non-expert text coding.

      Drew's a leader in the New York data scene, so we were already excited to have him, but the fact that his work focused on Amazon's Mechanical Turk service got me really excited. 

      In his talk, Drew goes into detail on how he used the platform to determine whether non-experts could properly identify bias in political text. Drew's a great speaker and this presentation was no exception --the video is a must-watch if you're interested in pursuing large scale research on the web.

      Thanks again, Drew!

      1 comment
    • Kickstarter meetup at RubyConf


      Join Kickstarter engineers @tiegz, @emilyreese, and @ktheory at RubyConf in Miami Beach.

      We're hosting a meetup and drinks at the Segafredo l'Originale bar Friday, Nov 8 at 8pm. We'd love to hang out and talk shop; whether that's crafting mature web applications, inspiring OSS projects, or your latest DIY 3D printer hacks.

      The Details:

      Leave a comment
    • Unit Testing for Rails Views

      Sometimes, view logic happens to good people.

      It might be checking for empty result sets. Or rendering optional related data. Sometimes it's state- or role-based logic. It happens.

      A typical Rails app tests the view layer during functional controller tests. But this coverage feels accidental at best and inefficient at worst. We wanted better coverage with faster tests that people would actually want to write and run.

      So we split apart our functional tests. Here's how.

      A View Testing Pattern

      First we experimented to find a good pattern. We extended ActionView::TestCase and worked out a 3-part syntax based on shoulda.


      class ProjectsViewTest < Kickstarter::ViewTest
      
        context "show" do
          setup { @project = FactoryGirl.build(:project) }
          subject { render 'projects/show', formats: [:html] }
          should "render" do
            assert_select '*'
          end
        end
      
      end
      

      To make this work, we needed to ensure that subject was triggered:


      class Kickstarter::ViewTest < ActionView::TestCase
      
        # shoulda lazily evaluates subject and then memoizes it.
        # so we just need to reference it. the result of any
        # `render` in the subject block will be available for
        # assert_select thanks to ActionView::TestCase.
        def assert_select_with_subject(*args, &block)
          subject
          assert_select_without_subject(*args, &block)
        end
        alias_method_chain :assert_select, :subject
      
        # ensure a default subject_block
        def self.subject_block; proc {} end
      end
      

      One nice part about using subject this way is that we could easily test variations with nested contexts:


      class ProjectsViewTest < Kickstarter::ViewTest
      
        context "show" do
          subject { render 'projects/show', formats: [:html] }
      
          context "for a live project" do
            setup { @project = FactoryGirl.build(:project, :live) }
            should "render" do
              assert_select '*'
            end
          end
      
          context "for a successful project" do
            setup { @project = FactoryGirl.build(:project, :successful) }
            should "render" do
              assert_select '*'
            end
          end
        end
      
      end
      

      Transitioning

      Once we had a syntax we liked, we presented it to the team and talked about how to proceed. We wanted to write new tests this way, but back-filling the suite was daunting.

      So we set our sights on a realistic goal. The current controller tests had some existing accidental coverage. Why not just move it?

      First, we identified the existing coverage using loggers patched into the controller's render method. This generated a list of templates that looked lengthy but doable.

      Then we created an ERB template to auto-generate test stubs for all of these templates. Some of the auto-generated test stubs worked right away, so our list immediately got shorter.

      It took a few days to finish out the test suite. This gave us time to think about our patterns and set precedents for the rest of the team to follow. In some cases we even got to refactor a bit of code so the tests would be easier. Bonus!

      Disabling Controller Rendering

      Once we had a suite of view tests, we decided to disable controller rendering. The trick to accomplishing this was in supporting assert_response and assert_template without messing up the template compilation that we needed for the view tests. Here's the simplest patch we came up with:


      class ActionController::TestCase
        setup do
          ActionView::Template.class_eval{ alias_method :render, :stubbed_render }
        end
        teardown do
          ActionView::Template.class_eval{ alias_method :render, :unstubbed_render }
        end
      end
      
      class ActionView::Template
        alias_method :unstubbed_render, :render
        def stubbed_render(*)
          ActiveSupport::Notifications.instrument("!render_template.action_view", :virtual_path => @virtual_path) do
            # nothing but the instrumentation
          end
        end
      end
      

      The payoff was worth it. We cut about 35% off our controller suite time, and since we used build and build_stubbed strategies for the view test factories, only a fraction of that time was spent running the new view tests. Nice!

      In Review

      Our test suite feels more focused, more performant, and easier to extend. We have more confidence in our ability to make certain refactors in the views layer. And we can still rely on integration tests to put all the pieces together for critical paths.

      We're continuing to explore the pattern, but so far this just feels right.

      1 comment
    • HTML5 Video First

      Video is an integral part of Kickstarter, and ever since our launch, we’ve depended on Adobe’s Flash to serve videos in a proprietary player. Today, thanks to HTML5’s <video> tag, that changes.

      Previously we had been serving our project videos to desktop machines using Flash. If Flash was not detected, our players would fall back to a HTML5 <video> element. 

      Starting now, however, we are inverting this logic and will only serve Flash videos if users' browsers are so old they don't support the <video> tag (including some versions of Firefox that don't support h.264).

      Why

      Most mobile devices do not support Flash. It’s simpler to use the exact same software on both the desktop and mobile.

      Some computers do not ship with Flash. We don’t want to require users to install software to use our website. We still use Flash for other features (users have to use it to upload media), and we will work to remove those Flash requirements.

      We have never had an in-house Flash developer, because while our video player is important, most of the client-side code is written in JavaScript. Every time we have wanted to redesign our Flash player or add a new feature we have had to ask an outside consultant, which took time and made quick turnarounds difficult.

      Flash is insecure. Due to ExternalInterface.call, it’s extremely easy to accidentally allow the execution of Javascript sent via a param.

      How

      For a couple weeks now, we’ve been building up our own HTML5 player. We started serving our new player to people who didn’t have Flash installed, which allowed us to work out the UX and bugs with a small set of users. A few days ago, we enabled the HTML5 player for employees of Kickstarter, widening the user base even more.

      We haven’t found the need to use a larger library because the video element specification is so clear and simple to use. Our designers are very skilled in CSS, and we used those skills to design the look and feel of the video player. It is a pleasure to use our core abilities on something that is so core to our site.

      Who

      Design by Zack Sears and Brent Jackson.

      Interaction by Samuel Cole.

      6 comments
Loading small 9cd608b53c63844322bca1d7d2cfa9d9cf2b2d91b09deb1c37b02bb990161eab
Please wait