Video and Slides: API Metrics - What to Measure »

Thanks to all that attended last week's API Best Practices Webinar #4, API Metrics - What to Measure (and thanks to our presenters @brianpagano and @landlessness). Video and slides are below.

Our next API webinar, "Is your API Naked? API Technology and Ops Considerations" with @landlessness and @gbrail, is June 14th at 11am PST (sign up here!)

Moving the needle: Example API metrics »

It's an old cliche, but it's been said that you can't move the needle if you can't *see* the needle.  So frequently we're asked "what are good metrics to measure an API program?" 

While individual metrics are important - it might be as much about the 'process around metrics.'  Or..how metrics are evangelized and used to drive specific parts of the API product development pipeline.  Specifically:

1. Get early buy-in on the 'top 3'  -  strong API product managers often focus in on 1-3 top level 'strategic' metrics and get early, wide agreement from all parts of the extended team - the sponsoring exec, PM, engineering, BD, and operations. If different stakeholders are measuring success with different metrics (say number of developer sign-ups vs. API traffic vs. revenue) this can pull resources in different directions.  

2. Track against realistic projections.   Set expectations early by modeling anticipated results and then track actuals against this estimate. For example, pick a 'comparable' or competitor's API to guess developer portal traffic, then model the expected developer sign-ups and conversions  (for example, 10% of visitors might ask for a key, 20% of them might built an app, 10% of those apps might drive ongoing traffic, each of those apps might drive a certain volume of traffic, and so on... )  

3. Publish a weekly dashboard, religiously.  Proactively call out how product updates and community activities do or don't move the needle so you can quickly adjust tactics and think of new ideas that might move the needle.

4. Create a metrics 'pipeline' -  How do different metrics diagnose how each stage of the customer conversion process is working?  For example, developer portal traffic might be a good metric to measure the marketing guys. (that is, they might be responsible for getting developers *to* the portal.)  But whether or not a developer converts to ask for a key and then converts into an active API user might be a measure of how effective the PM process is working to create a product that developers want to use.  User experienced bugs can measure development and product QA effectiveness, and so on..

Here is an example of a metrics pipeline that we recently discussed with a customer.

(Thanks to seenoevil for the photo)

Tech Talk: API Visibility and Metrics »

Earlier this week, Greg speculated that Twitter might have benefited from digging deeper into API metrics and usage patterns, so we thought it would be a good time to put him on the spot with a tech talk he recorded on API visibility a couple weeks ago. 

For more, here are some sample API metrics considerations and a demo of our own API Analytics solution.

Is your API naked?  10 API roadmap considerations (part 1: visibility) »

We'd like to run a short series on product and technical requirements you might consider for an API roadmap or strategy.    We’ll base this on trends we see in talking to hundreds of product and engineering managers that are either opening or consuming APIs (or aggregating and publishing large numbers of RSS feeds.)

Instead of talking about your APIs functionality, this is about what's *around* the API features and data.  Many APIs start out as just raw naked back end features. And there is often a big gap between a raw feature and a full-fledged service, which is your API might eventually become.

So this series is about what's needed to "monetize", "productize", or "operationalize" an API.  And not just if you are providing an API to customers – much of this applies if you are consuming APIs as well.

Topic #1: API Visibility

We're always surprised how almost every company we talk to says how little they know about their API traffic and usage.   We see lots sifting through web server logs to understand usage. As the API becomes more strategic and you want to make the case for more investment - this gets more painful.

This happens a lot when an API starts as an experiment, launched by the 'ask for forgiveness, not permission' types (you know who you are).  Things like metrics or analytics are back burner until an API either gets off the ground or doesn't.

But most APIs usually end up getting more important more quickly than expected, and as a product and engineering manager you may start asking:

Who is using the API and how much are they using?

• How many clients, apps, developers are out there?
• How do they break down by type, region, class of service?
• How does usage map to existing customer or partner organizations. Or how do developers map to applications map to customers?  (This can be tough with only key or IP based tracking.)
• What parts of the API and service are they using - on a method or operation level?   
• How does traffic breakdown between the your own products and 3rd party products?  (If they use the same API.)
• What the aggregate and per developer/app/customer transaction and data volumes?

How fast and 'good' is your service? 

• What latency are customers experiencing, by developer, customer, region, or operation?
• Where are errors and user experienced bugs happening and how often? 
• How is the API delivering vs. any formal SLA have agreed to or paid for?
• How can you find out if a customer is having a problem (before they call you)?
• How is the API usage impacting the rest of the platform or web products that also use the same API?
• Can you quickly trap and debug based on a specific message? Based on what is in a cache? streaming right now?

How does the API impact the business?

• Who are the top priority customers?  Developers?   Partners?  Who should you call to up sell to a higher service tier or do a deal with?
• What do you need to show general management to make product strategy (or tactical) decisions?
• Will you need to create audit trails or metering reports for partners that are paying for API access?
• Do you need to create metrics based on a certain data value in the payload? (Such as a specific product SKU)
• What is the cost of the data that you are serving up? (if you are licensing this data)
• Are you in line with all the compliance standards that IT enforces for the on-premise apps?

Knowing this stuff is really important when opening an internal service as an API, because now customers, contract terms, and compliance regulations come into play.  Analytics and metrics help you get proactive with customers and partners, and are critical when you need to make the business case for an APIs to executives.  You probably use web analytics to help you improve your Web UI - at some point you need this for APIs as well to see where to invest.

What's your experience?  We’d love to hear what you think.  Next up: traffic management.