OAuth: The next Big Thing in Security »

Today at #defragcon our @sramji gave this talk making the case for OAuth as a business imperative.

(As Sam mentions on the last slide, no developers were harmed in the production of this presentation.) 

Trust, Safety, and the OAuth API »

At Apigee, we've been developing a secure, robust API management platform since 2005 and have been running it in the cloud since 2008.  We're proud that some of the most demanding enterprise customers like Netflix, Comcast, and GameSpy have made Apigee technology part of their API platform.

We believe that it should be easy to start working with APIs, so last week we launched a new service called the OAuth API, which takes the pain out of getting up and running with APIs that use OAuth.

The story of the OAuth API is the story of yet another function that we used to think of as occurring on a device but that is now moving into the cloud.

It marks the beginning of a trend of APIs being manipulated before they get to the client, since some things are better handled as a service. Authentication mediation is one of those things.

Apigee has built a business of using our API Gateway to publicly expose backend APIs in a safe way, such as by adding OAuth as the mechanism by which applications can be granted permissions by end-users of applications.

The OAuth API takes that model and reverses it, taking something that is complicated for developers to implement (especially across multiple APIs) and simplifying it.

But is it secure?

Should you trust your OAuth keys to a cloud provider, and to Apigee in particular?

In many ways, the cloud might be more secure, since providers do things you might never do on your own. Far too many applications never bother with encrypting users' secrets in their own databases. When you use the OAuth API, you get that for free, in addition to the API normalization and long-term maintenance wins. 

If you do trust the cloud, should you trust Apigee? We respectfully submit that it comes down to three things: technology, processes, and people. 

Technology

Our technology processes billions of OAuth transactions a day. Our Apigee Free tools—including the OAuth API—are built upon this same platform that powers 250 enterprise customers.  It is a core requirement that our servers be both security hardened and highly available.

Processes

We encrypt the secrets entrusted to our database. We run security audits. We involve our security team members throughout the design and architecture process.We use the same software, made by the same people, working under the same processes behind the products that power the APIs of the most demanding Fortune 500 companies. We use industry standard security practices.

And in the end, it's still an OAuth token that is being stored, not a user's password. It can be revoked by a user at any time, and developers can still invalidate their consumer key and secret at any time.

People

Here at Apigee we have lots of experience in API security and operations in the cloud, again as a result of working with demanding enterprise customers for years, many of whom entrust us to protect their API traffic, their organizations, and their customers.

Q&A for OAuth and API Best Practices Webinar »

Thanks to all that attended last week's webinar:  OAuth: The Big Picture (slides and video here).

There were great questions  - here are thoughts on those we didn't have time to discuss.

Q: Would you recommend only using oauth for passwords and usernames?  What about more confidential items like card numbers? (John G)

 I'm not sure I completely understand - but are you asking - if a user can authenticate to OAuth using a credit card number rather than a username/password?

That's theoretically possible but I'm not sure it's a good idea. Also, keep in mind that most credit-card authorization systems use a unique ID to map your transaction to an actual credit card, and for PCI compliance the actual number itself is stored in a PCI-compliant card system. In other words,  I'm not sure that OAuth would be solving a new problem here.

Q: Can you talk about the practices around validity periods of tokens, given that their decoupled from password validity? (Eve Maler)

 Sure. First, why do OAuth tokens need to be refreshed at all?

For one, when an OAuth 1.0a or 2.0 "mac" signature is used without the benefit of SSL, it could theoretically be cracked given enough time and computing resources. However when SSL is in use, you can't do this by just sniffing the network, although if you can intercept a request some other way then you could theoretically do this. So, one reason to refresh the token is so that it can't be cracked.

Another reason is just because as users pass their computers around, phones, etc, it's possible that they won't realize that they contain OAuth tokens. This is a much more subtle and undefined reason than the first.

Some applications give out tokens that never expire, which leads to the theoretical "cracking" example if you are not using SSL. On the other hand, I've seen applications with tokens that expire in 15 minutes, which is not very user-friendly. I don't have a specific recommendation right now and I'm not sure if I have seen another as well.

I actually think that this is a good area for some new research if there's someone out there looking for a project -- I don't think that we've quantified this well.

Q: If the user is entering the user/pass on the browser, even though it may be a trusted website URL, it is still vulnerable for some client to sniff the credentials as it is on the machine itself!! (Kaushik)

 Sure -- nothing on a computer can be completely trusted, and the person operating the machinery can be trusted least of all!

What OAuth does for a web app, is keep the passwords from the server that you are using and prevents your password from being spread around the Internet from server to server. Given the number of security breaches in which servers are compromised and user accounts exposed,  this fixes a big problem.

What OAuth does for a mobile app is keep the password from the application that you are running. I think you can trust the browser embedded in your OS more than you can trust the application built by a developer that you don't know, but I know that others here disagree. (And yes, I realize that an app can put up a fake browser screen. I wonder if such an app can get through the Apple App Store? Does anyone have knowledge about how real a threat this is?)

Q: What options does Apigee support for server-to-server API calls where we want to verify the caller is a valid caller?  i.e. request signature? (Mark Visosky)

 We can support two-way SSL, WS-Security with UserNameToken profile or X.509 profile, SAML, XML encryption / digital signatures, and of course HTTP basic authentication.

Q: if i change my password on twitter, then try to use a previously working app, it makes me reauth (Daniel)

 I didn't know that about Twitter and should have tried before the call.

Q: Thanks for the webinar, you've clearly thought a lot about authentication and security issues.  However, I cannot grant you the point that opening up a web browser where you enter your password to the service provider keeps the user's password secure.  A malicious desktop or mobile application will be able to trick the user into divulging their password by spoofing the oAuth page in some way. oAuth sign in page, I mean (Kevin)

Thanks for the questions. There are certainly other ways to trick a user into divulging their password even beyond what you suggest... That doesn't mean that we have yet abandoned passwords completely.

However, an API provider that DOES want to abandon passwords and replace them with some sort of multi-factor authentication, VPN, SAML, hardware dongle, whatever, can do that by replacing the authentication screen in OAuth with one that requires whatever that API provider wants WITHOUT requiring any change at all on the client. This is a lot harder to do once there are clients out there that have hard-coded username/password authentication.

Q: if it again asks for reauth then does that mean it is using the changed password somehow..? (Kaushik)

An API has the option of revoking the OAuth tokens at any time, and some may choose to revoke them whenever the password is changed. However, OAuth doesn't require or specify that.

Q: Isn't the full OAuth dance only necessary if a user needs to authorize another application to perform actions on their behalf?  i.e. giving authorization to bit.ly to submit tweets through my twitter account.  What about API's where we just want to validate callers are who they say they are but no end-user is involved. (Mark V)

 For server-to-server communications, two-way SSL, or "regular" SSL with HTTP Basic and a long, random password are fine ways to solve this IMHO.

For applications that need to authenticate themselves as "the app" rather than as a particular user,  you can assign each app a unique and random "application key" and include that in each request. Use SSL as well if keeping the app key a secret is important to your business.

Q: Do services need to store/control which third-parties can retrieve your OAuth credentials? Or is it generally the rule that if you expose OAuth, any app can use it? (Kenneth)

 In order to build an application that uses an API via OAuth, the application needs a set of application credentials -- that's the case for OAuth 1.0 and 2.0.

Q: i want to write an OAuth2 *provider* - what libraries are available to me? (Kevin H)

There is a long list of open-source projects at oauth.net under "Code." It's not something that can usually be just "plugged in" to a server, however, because it depends on how you want to authenticate users, where you want to store credentials, etc. At this point in time I should mention that an API gateway product such as Apigee Enterprise is one great way to get it done quickly.

Q: If oAuth requires SSL in 2.0, the request signing step becomes useless, no? (Kevin W)

When OAuth 2.0 is used with a "bearer token," then there is no secret and no request signing -- you use SSL instead.  When OAuth 2.0 is used with a "Mac token" then it is just like 1.0a.

Q: Are the specs for SSL ,SAML assertion available to be used or implemented for my server..? (Kaushik R)

Absolutely. Check ietf.orf and search for "OAuth." There are active groups working on all these specs.

Q: If you're requiring SSL, why not just trade un/pw for a token set and be done with it?  This assumes that the oAuth sign in page is security theater, which I maintain that it is. (Kevin)

@Eve but you can use any RSA spec, as long as the client and server know what to do. (Dan)

un/pw/api key or whatever? (Kevin)

Even the father of OAuth, Eran Hammer-Lahav, says using SSL as a means of authentication is broken (ie, OAuth 2.0 is broken) - (Dan)

 Yes, he is wary of SSL.  That's why he personally ensured that OAuth 2.0 also includes the "mac token" option, which is a way to use OAuth 2.0 with a signature and without SSL, just like 1.0a. This particular spec is newer and changing more quickly but it'll be an important option soon. Given the amount of work he is doing on the spec I don't think he thinks it's broken.

Q: How complex it is to implement OAuth in our own servers..? (Kaushik)

 The hard part is integrating the OAuth flow with your existing authentication mechanisms, so that the browser is redirected and all that. There is also a bunch of work involved in implementing the various authentication flows, and there are quite a few of them. I think there are much more complex things to implement but OAuth does have its challenges.

Q: Where is the OAuth token stored? is it Server side in your example Twitter, or client side BIT.LY or BOTH? I think you mentioned in case of mobile apps it is stored on mobile. (Arpit)

 It depends who is accessing the API on your behalf. The bit.ly web site, for instance,  must store them on the server since it's a web site. The Twitter mobile apps, however,  store it on the mobile device itself.

Q: So are they saying that banks shouldn't use OAuth for banking transactions? (Kenneth K)

 If a bank has a fiduciary responsibility, for example, to sign every transaction with an RSA key that is stored on a hardware key store and managed by a PKI with tight policies and procedures around key distribution, then OAuth isn't going to be used in such an environment any more than plain old username / password combinations. That's the case for financial applications that deal with individual transactions worth millions or billions of dollars.

Seeing that many of us bank on our mobile phones or web browsers using just a username and password for authentication, I think that OAuth is just fine for a wide variety of other financial apps.

Q: The oAuth login screen is security theater - I can spoof it or otherwise trick the user if I am a malicious mobile or desktop app developer.  If you can't force me (the third party app developer) to NOT store the un/pw, why make me goof around with an embedded browser page in my desktop/mobile app?  Thoughts? (Kevin)

 I think I tried to answer this elsewhere, but in general if you really want to do this then OAuth 2.0 actually includes an authentication flow that lets you get a token by simply sending the username / password to the server.

Q: oAuth gives a false sense that your transactions is secure, but the spec really does not mandate any message level security?? (Praveen)

 Well the spec strongly recommends it but the spec can't force it -- it is up to each individual API to require message-level security in the right places.

Q: Advanced talk should have a topic on usage of the "SCOPE" parameter (Vladimir)

I think that the whole "scope" thing would be an interesting seminar,  yes. Let's consider it,  but we'll need a lot of good examples!

Thanks again to everyone and hope to see you on our next webinar:   "Boss, we need an API!"

Race conditions: Does your API TOCK TOO much? »

We've written on the topic of API threat detection before, and also outlined a "top 10 API threats" to guard against, but race conditions are another area in which APIs are also vulnerable.  

A race condition is a bug where the output is dependent on a sequence of timing of other events.   APIs are vulnerable to a type of race condition called TOCTTOU (pronounced "TOCK TOO") .   

During this crack in time, malicious users are using race conditions that have been exploited as security vulnerabilities in systems for almost 4 decades.

Historically TOCTTOU attacks have been used to take advantage of a weak filesystem API and limited security controls to OPEN, RENAME, CHANGE OWNER(chown), CHANGE MODE from READ ONLY to READ/WRITE (chmod). Applications such as vi, gedit, rpm, emacs, gdm, and many other GNU programs have been exploited to take advantage of TOCTTOU weaknesses. (See this even longer list at the NIST CVE CCE Vulnerability Database.)

What can you do to reduce race conditions in APIs?

Mediate the request and response with a single atomic event that collapses the check (authenticate) operation with the use (open) events.  If you are building atomic transactions as the design principle, then buffer the system with a checkpoint between the user zone and the service zone — aka, the DMZ.

The use of a mediation proxy in the DMZ to perform the authentication and authorization checks adds latency indeed, but it improves the quality of the system by adding load balancing, packet and message body introspection, event logging, and possibly a second authorization on the response to verify that the user is authorized to receive the object requested.

A second authorization check can also:
-    prevent accidental data leakage
-    makes it more difficult to create a race condition
-    detect if the user is no longer authorized to read the content, or if the user is attempting to access an object without execution, read or write entitlements.

So if you are concerned that your firewall is not doing enough for you, or you are worried that your DMZ has been compromised by the firewall TCP Split Handshake vulnerabilities, then consider using an extra layer of mediation between client side applications and server side APIs.

A policy enforcement point (like our Apigee API Gateway)  - behind the firewall and in front of the API service - reduces the risk of a man-in-the-middle attack.

Are all OAuth implementations Created Equal? »

We've implemented a couple dozen OAuth implementations in the past months, and no two have been the same.

We've seen Oauth 1.0a (2-legged,3-legged), Oauth 2.0 (username password, user-agent, web-server flow) with custom token types (bearer,mac, saml).  Just to name a few.

We wanted to write some posts that discuss factors we see that drive these variances and their implications.  We also want to compare and contrast each of these variances along with our own lessons learned. 

First, why the variances?  In our experience, the primary cause of the variance is the type of application that the API is targeted for, which tend to break into one or more of 3 types.

Partner Apps - typically a "closed" API available only to a select group of partners with whom you collaborate very closely to build the apps. For instance - an health insurance provider who is exposing an API that a partner can use to build some health related activity tracing applications.

Open - Browser / Mobile Apps - APIs are exposed to a large community of 3rd party developer from whom you are expecting a large scale of applications to drive innovation. Here you do not control who, how many and what kind of applications get build using the APIs.   For example, mobile applications that integrate with the social networking sites.

Enterprise Apps -  when the API is exposed for use primarily by other enterprise customers - much like a B2B scenario - to allow the customers the freedom and flexibility of building in house Applications and services that mashup one of more of the applications from multiple vendors for internal optimizations.  A good example could be an enterprise that wants to use the Sales Force API along with one other internal apps to build an internal collaboration app.

Depending on the application type above, your approach to an OAuth implementation may vary:

An API for Partner Apps will typically follow a “User name / password” flow.  In this case the end user’s  user name and password will be passed from the user to the partner app and the partner app transmits this to the origin / backend API. The key consideration here is that the partner app will have direct access to the end users credentials and is trusted to treat this information sensibly.

In the Open Browser / Mobile apps scenario the app itself is not inherently trusted and is not allowed to handle the end user credentials directly. The onus of authenticating the end user falls squarely on the origin / backend API provider. The provider (after the end user authentication is successful) will simply issue an access token to the app – this token will correspond to the set of capabilities that the end user has authorized the application for. The sequence of steps followed in this case is commonly referred to as the “user-agent flow” model.

The Enterprise Apps scenario is by far the most robust flow (and understandably, the most complex one too). Here the access token is given to an intermediary server instead of the client device and this facilitates a more secure transmission of the access token. In this scenario, it is possible to restrict the source from which the web service is invoked. This sequence of steps is commonly referred to as the “web server flow” model.

Sounds complicated?  It is. We will try to simplify each of these variants and throw some more light in a series of follow-up entries. Stay tuned!

Why you need OAuth for your API or App »

I've been posting a bit on OAuth best practices recently. But I want to take a step back and talk about why OAuth is important in the first place, why it's difficult, and how to think about it for your API.

Why is OAuth important?
 
First, OAuth supports delegated authentication between web apps. That means that if web app X offers an API, and web site Y wants to use that API, then no one needs to type their "X" password into a screen controlled by "Y." This is a big deal and it's being used by a large and growing number of apps.
 
Second, OAuth does this using a security token called an "access token." An OAuth access token is a unique identifier that allows whoever has the token to get access from one and only one application to one and only one API. (The creators of OAuth describe this access token as a "valet  key," and that's a good description.)
 
An access token is different from a password, which allows whoever has the password to get access from any application to an API, because it is tied to a single application. Plus, access tokens are large and random, so they're not subject to a dictionary attack like a password, and they are hard to share. It's not going to be the case that your access token for Facebook is going to be the same one that you use for Foursquare, whereas if you're like most humans you might use the same password!
 
Third, in order to get access to an access token, an application itself must somehow be authenticated. That means that not only do users need security credentials in an OAuth world, but so do applications. This is a good thing because apps can't always be trusted, and with OAuth each API provider has the ability to revoke access to a rogue application if necessary. (Although there's no way to make it totally impossible for a determined hacker to extract the credentials from a mobile app given enough time and effort, the existence of these credentials adds a layer of security that wouldn't be there otherwise.)
 
Fourth, OAuth makes it possible to separate the assignment of API security credentials from the process of authenticating users to a web site. What does that mean? It means that if you are big company with a web app, and you want to offer an API, it's not necessary to wire the API into the web site infrastructure. Instead, the API can be implemented alongside the web site, and only link to one web page for the actual user authentication. (And even this can be done in other ways.) That means that companies of all sizes can build OAuth-enabled APIs without impacting their existing applications, and that's a good thing.
 
Getting all these things right is hard. For instance, the very first "production" version of OAuth 1.0 didn't actually do the first thing -- authentication delegation -- correctly and as a result the spec itself had to be patched (that's why we talk about OAuth 1.0a now). A lot of people looked at 1.0 without realizing this, which goes to show how tricky it is to create a new security standard from the ground up without making big mistakes.
 
Why has OAuth been difficult?
 
Right now, the OAuth world is sadly full and ambiguity. This is largely because of implementation work on the OAuth 2.0 spec, which offers the promise of an easy-to-use OAuth specification, but at the same time keeps changing. We fully expect the rate of change to slow or even stop in the next few months, but no one knows for sure, especially since it is an IETF process, which takes time.
 
However, it takes time to do a job right, and security standards that impact the whole Internet certainly deserve to be done right.
 
Unfortunately, that puts today's API implementors between a rock and a hard place. Until OAuth 2.0 is a little more stable, you have the choice of implementing the latest draft and being prepared for changes, and implementing OAuth 1.0a, which makes life more difficult for customers.
 
What should you do?
 
Be patient. It's possible to build an effective API right now using either a current draft of OAuth 2.0 (see Facebook) or OAuth 1.0a (see Twitter).
 
However, things are going to change a bit for the near future. Some would take this as a reason to give up, "sunset" OAuth, or declare it "dead" way before its time. I think that building a new security technology from the ground up that has all the advantages I enumerated at the top of this article would be a big mistake.  We have two pretty-good options today in OAuth 1.0a the latest drafts of 2.0 -- stick with them.

Pragmatic PCI Compliance and APIs: Just Enough Process »

“If the minimum wasn’t good enough, it wouldn’t be the minimum.”  - Keith W.

Wise words from one of my developers many years ago.  When it comes to tackling PCI Compliance, it is advice well worth taking.

With leaks of sensitive customer information in the news, there’s an increased focus on compliance as more services shift to cloud computing and APIs.  

If you are a merchant of any kind or deal with customer credit card information then you must be aware of PCI compliance regulations that are designed to protect consumer credit card information from exposure.

PCI compliance gets tricky as apps and services move to cloud services and APIs.   If you’re heading down the path of PCI compliance or just trying to position yourself, your APIs and your internal systems better for the future, keeping it simple will help you be successful.

First, Document your Process

The PCI Data Security Standards (PCI DSS) establish the “what” but not the “how” of achieving compliance. 

The how is up to you. But like most audit and process centric assessments, what is most important is being able to articulate what you do to support a particular DSS item - and then being able to show evidence to support that statement.

Identify all of the process standards that apply to you from the DSS and identify the proper owner of those processes as well.  Put together a simple Process Description Template that everyone uses to document their individual processes and adopt a naming convention that calls out the DSS section.  Centralize the storage of those documents and make sure everyone knows where they are. 

Just focus on capturing the “how” of your processes in as lean a manner as possible.  Your assessment team is not going to evaluate quality of the process or the documentation, only that it meets the requirements of the DSS.

With your processes documented "well enough" and easily mapped to the PCI DSS, you’ll discover gaps, strengths and you’ll make your assessors life easier and that makes your life easier.

Next, we'll talk about the special challenges with PCI in cloud computing and APIs, and practices that you can apply to reduce your risk.

Morgan Hall is a project manager with Apigee. Previously he was Director of Business Architecture at TransUnion.

Best Practices for OAuth 2.0 vs. OAuth 1.0 - One year later »

Since we first wrote about OAuth 1.0 vs. 2.0 a while back, lots has been happening and lots has changed. The OAuth community has made progress and made changes, and an increasing number of API providers have deployed APIs that use OAuth 2.0. (Similarly, the number of new OAuth 1.0-enabled APIs doesn't seem to be growing.)

We've been busy helping our customers implement OAuth-based APIs, and we've also been watching the process develop. Here are some things that we've learned: 

OAuth is a solution -- not technology.

In order for an API provider to support OAuth, a number of things have to happen. There must be a place where the OAuth protocol is actually implemented -- all of the different URLs for requesting tokens, issuing tokens, redirecting browsers, and so on. This part is a matter of implementing the standard and doesn't change from implementation to implementation (so, it can be delegated to a third-party component like Apigee Enterprise).

In addition, there has to be somewhere to store OAuth tokens. There is no standard for this -- some use an RDBMS, some use NoSQL, and others use a cloud-hosted service.

Finally, the OAuth solution has to integrate with a provider's "login page," so that when a user wants to get an OAuth token, they are redirected to a legitimate "login" page with a proper SSL certificate,  look and feel, and so on. All of these kinds of things can be different from one provider to another.

At Apigee, we have been working with many companies to deploy APIs built using OAuth. What we've found is that OAuth is not just a technology that you buy or download -- it's a bit of technology and a bit of integration, and to implement it at all you need a platform that can flexibly integrate with what you already have, while adding what you might be missing. 

Be careful.

Since we last blogged about it, OAuth 2.0 has been improved a bit and has changed some basic things -- for instance the name on the HTTP "Authorization" header has changed from "OAuth" to "Bearer." This is a big change, and obviously if an API provider just switches from one value to another without warning, all the clients will break.

OAuth 2.0 is up to draft 13, and with every draft it gets closer to being "done" -- but it's not done yet and I haven't seen anyone say when it will be. That means that if you implement, say, draft 13, you will likely have to change, and find a way to maintain backward compatibility for existing clients.

If that's too much, then implement OAuth 1.0a -- it's as "done" as it's going to get at this point! 

Stick with "bearer tokens" for now in conjunction with SSL.

The latest draft includes support for various types of security tokens,  and a mechanism to plug in new types. The simplest are "bearer tokens." A bearer token is just a big, random string that a client must present on every API call. Bearer tokens are simple because there's no special signature or validation code required on either end. The client is responsible for storing the token in a safe place and sending it with every request. The server is responsible for looking up the token in a database and making sure it's a valid one -- that's it.

However, bearer tokens provide no security unless used in conjunction with SSL -- otherwise, any eavesdropper in a coffee shop can snatch OAuth tokens from  the air. (Fortunately, OAuth tokens can be individually revoked without requiring a password reset.) 

Get ready to change -- or stick with 1.0a.

We said this before but it bears repeating -- if you implement OAuth 2.0, it is likely that you will have to change your servers to support newer versions, and also maintain backward compatibility.

One way to ease the pain is to put version numbers in the URLs that you hand out for the various steps in the OAuth flow. So, instead of using:

http://api.foo.com/oauth/authorize

use:

http://api.foo.com/oauth/13/authorize

because you know that there will be future versions and that they will be different...

Get ready for more options.

 Part of the OAuth 2.0 work defines some new security token types other than "bearer." Right now only "bearer" seems to be sufficiently mature from a spec standard, but this will change over the next few months. Here are some of the options:

1.       Bearer -- as we mentioned before, this is just a long, random string. The advantage is that it's extremely easy to implement on both client and server. The disadvantage is that it requires SSL for all API calls. Also, if API traffic passes through multiple endpoints such as proxies, then the proxies will be able to see the token as it passes by.

2.       Mac -- this is equivalent to the token scheme in OAuth 1.0, in that it requires both a key and secret, and uses an "Hmac" algorithm to encrypt some data on each request. The result is that the request is only valid if both client and server have the same keys, and there is no way for an intermediary to re-create the original request without a password. In other words, it is secure even if SSL is not used or even if the request is intercepted by a proxy.

3.       SAML -- this uses SAML assertions on each request as a way to establish the client's identity. This is helpful if you already have SAML infrastructure for either the client or server in your environment.

A Short History of API Authentication (and where it’s going): from HTTP basic to OAuth 2.0 »

Part 1: The Web

In the beginning -- way back in the beginning -- the web was all about open access. Tim Berners-Lee and his colleagues focused on making information available, not on protecting it from unauthorized users.

But as time went on, and as Al Gore took the initiative in liberating the government-run Internet backbone for commercial use (really), the Web became about "e-commerce,"  and e-commerce required security. SSL matured to ensure that sensitive traffic was encrypted all the way from the client to the server and back, and various schemes emerged to allow user authentication.

The oldest and most common format for web authentication is HTTP Basic authentication. This is what you get when you visit a web site and a little browser window pops up requesting a username and password. Every web browser and every major web server supports some form of this.
 
From a web design perspective, HTTP Basic has a big disadvantage in that it's implemented entirely by the browser, and can't be customized for each site. As the quality of graphic design improved on the web, designers soon realized that they wanted not a generic little grey box on the screen, but a carefully-designed login page, with logos, disclaimers, and the like, or a discreet login button on the corner of a web page. The combination of HTTP Basic and HTML just didn't allow this.

The result was the rise of form-based authentication. This is what the vast majority of secure web sites use today. As a user, you visit a web page that prompts you for a username and password. If authentication is successful, then under the covers you are granted a unique cookie, which your web browser sends with each subsequent request. As the user you never see this -- it just looks like you logged in and now the site works -- but under the covers it is quite different from Basic.

Both Basic and form-based authentication rely on SSL to create an encrypted tunnel between the client and server. Without that encrypted tunnel, anyone snooping the Internet or the open Wifi at Starbucks can see the passwords go by in the clear. Fortunately, SSL protects against this very well, but sometimes developers neglect to use it, users neglect to ask for it (as most of us do with Facebook), and sometimes the traffic travels over unencrypted links behind the firewall of a large network.
 
The web community attempted to counter this using HTTP Digest authentication, which encrypts the password using a one-way hash so it's impossible to reverse-engineer even if SSL is not used -- but it still must be implemented by the browser and can't be designed in to a nice UI. It never took off.
 
For a higher level of security, SSL has long supported two-way authentication, which requires that individual end users request digital certificates for each site they plan to visit and install those signatures on their browsers. The overhead of issuing PKI credentials to end users was enormous and this never took off either.
 
Part 2: Early APIs
 
Some early APIs were built right on top of existing web sites built using form-based authentication. The easiest way to implement them was to use the same authentication mechanism, so API developers would create a method called "login" that returns a security token, another method called "logout," and require the security token on every API call.
 
This approach makes the API easy to tack on top of an existing web app, but it is more work on the client side and hurts API adoption. An administrator can't as easily drive the API from the script without logging in, extracting the security token from the response, making the call, and then logging out.
 
Other early APIs just use HTTP Basic authentication. It's simple, works with every client (and with every shell script based on "cURL"), but it requires SSL to be used, often leaving it up to the client to "remember" to use it. Still, it's effective as long as the user has the password for the API handy.
 
Yet others, especially Amazon, decided they wanted to avoid using SSL for performance reasons, but they also wanted to avoid using the uncommon HTTP Digest authentication. (Amazon's S3 is used to store multi-gigabyte files and SSL does make a difference there.) They chose to create their own access control mechanisms based on secret keys and in some cases, digital signatures. The result was a bit of programming for each developer starting out with AWS, but Amazon's services were so useful and cheap that it didn't matter. By now there are numerous libraries to make this process easier.
 
Part 3: APIs get formal
 
The first real access control mechanism aimed at the needs of APIs and API developers is OAuth. The idea came from a popular API (Twitter) and a defunct web site (Ma.gnolia). The goal was to make it possible for a Ma.gnolia user to access Twitter without requiring that each user give Ma.gnolia their Twitter password.
 
The result, OAuth 1.0, is like a "valet key" for an API -- it is a token that gives a single client or web site access to a particular API on behalf of one user. The client or web site can get an OAuth "access token" without ever seeing the user's password because the two web sites do a sort of "credential dance" to exchange the secret token. Once the token is issued, the user can see it or revoke it, thus taking away access from the client or site without requiring a password change.
 
Since then, OAuth 1.0 has seen a patch (it became 1.0a to fix a security flaw), and an IETF committee is working on OAuth 2.0. This new version includes some important simplifications and a wider range of use cases. OAuth has become the de facto standard for API authentication. (for more see my earlier post on OAuth 1.0 vs. OAuth 2.0)
 
The Future
 
OAuth is likely to dominate the world of APIs for a long time. With OAuth 2.0, soon API providers and users will have the option of using a simple "bearer token" in conjunction with SSL, or a signed token that can remain secure without further encryption. OAuth 2.0 is flexible enough to be used in the original "web site to API" use case that OAuth 1.0 was designed to handle, but it can also be used for access by mobile devices, where it can be important to be able to remotely revoke the access tokens that might be stored on your phone without going around and changing innumerable passwords.
 
Still, there remain problems to be solved. What about access for simple management APIs, for instance? Even OAuth 2.0 is cumbersome and when SSL is always in use sometimes a plain old password is sufficient. API providers can and should consider supporting good old Basic authentication alongside OAuth 2.0 if for no other reason than convenience -- as long as SSL and a strong password are required.
 
What about mobile apps? Is there a way for the server hosting an API for, say, a large bank to ensure that the request is coming from an official application, or from a rogue app that is attempting to "phish" passwords by pretending to be the real thing? Can we do something by combining signed applications with server-side validation, or is a secure app store the only way to protect against mobile malware?
 
The world of APIs is evolving and there's no doubt that security technology will continue to evolve along with it.

(For more on API security issues see our other entries on OAuth and API authorization, identity, and security - or get it all with our "Is your API Naked" whitepaper)

Crossing the Streams:  Handling cross-site API requests (JSONP, CORS, UMP and more) »

My earlier post "Not using JSON and JSONP? You're doing it wrong!"  generated a lot of questions about the best ways to handle cross-site API requests from JavaScript running in the browser.

(Short intro -- the browser won't let you do it with the standard XmlHttpRequest API because of the "same-origin policy." This policy is in your browser's JavaScript engine so when you hit an application at http://www.foo.com, there's no way for http://www.badguys.com to return bad data to your application.)

This is an important topic, because if you are providing an API at api.yourapi.com, no one can use your API from inside a browser unless you implement one, and preferably more than one, of these techniques. It's also important because the state of the art is changing and changing fast.

Here's a run-down of the various techniques today, and how you might choose,  with the inevitable links to Wikipedia to help guide you: 

JSONP:

What it is: You return JSON from your API, and wrap it with a user-supplied "callback" identifier. This turns your JSON response from a JavaScript object into a function call or an assignment statement. Your user uses the DOM API to dynamically insert "script" tags into the web page to cause your API to be invoked, then executes the result as a JavaScript function call to create the object and start using it. 

Why it's good: Works on every browser, and asynchronously like every JavaScript network resource to boot. If users use a library like jQuery it is very easy to use.

Why it's bad: Only works with JSON,  although if you read my last entry then your API returns JSON anyway! It only works with GET, so you have to pollute your beautiful and philosophically-correct REST API by adding something like an "action" query parameter -- this tells your server that even though it just got a "GET" call, it should behave as if it were a POST, and you can't upload data with GET.

Finally, JSONP is really a hack around the security sandbox in your browser. As a result, it lets your JavaScript code download executable code from  another domain and run it inside the browser. If the user of your API trusts you, then that's great, but otherwise they're placing the security of their web app in your hands.

Aside from offering JSON as a response format with callbacks, you can also specify XML as the response with JSONP-X.

CORS: 

What it is: Cross Origin Resource Sharing is a W3C spec that allows a web service to specify that it's OK for it to be invoked from any domain. The API provider implements it by returning a special set of HTTP headers with every API response, and also by implementing an HTTP OPTIONS "pre-flight" request that lets the browser check first to see if a request is going to be OK. The actual spec is awfully complex but what you actually need to know in most cases isn't all that bad.

Why it's good: Because it lets the API consumer easily say, "it's OK, browser -- go ahead and make that API call." Once it's implemented, browsers that understand the CORS spec will automatically open a hole in the cross-origin policy as part of the standard XmlHttpRequest API -- the programmer of the web app doesn't have to do anything special. Plus, it can return any kind of data, not just JSON, and it works with all HTTP verbs. 

Why it's bad: It's pretty new and not all browsers implement it. Firefox 3.5, Chrome 3, and Safari 4 has it. IE8 does too but using a slightly different API. And support in many smaller mobile browsers (like Opera) is sketchy or nonexistent.

In other words, CORS is super-duper awesome as long as you don't care about people who run IE -- whether that's important to you is up to you

UMP:

What it is: UMP is a re-boot of CORS -- it handles 90 percent (or more) of the reasons an API would want to implement CORS without 90 percent of the complexity. It's also backward compatible in that it uses many of the same HTTP headers as CORS.

Why it's good: Like CORS, it allows an API to easily make itself available to web browser clients with a minimum of fuss. Better yet, all the API has to do is set a few HTTP headers, without also implementing an OPTIONS request.

Why it's bad: It's even newer than CORS. (New enough that it doesn't even have a Wikipedia page.) I have been told that recent versions of Chrome support it but it's not clear where else it can work at this point.

There's more to compare CORS and UMP here

Cross domain files

What they are: Flash and Silverlight also have a same-origin policy, but they do not work the same as the JavaScript engine and the techniques above do apply. What they do support are special files, namely crossdomain.xml (for Flash) and clientaccesspolicy.xml (for Silverlight). These function the same way as CORS and UMP headers in that they tell the client when it's OK to invoke the API across domains.

Why they're good: If you have Flash or Silverlight clients for your API, you need these too so that your API can be invoked from  those environments.

Why they're bad: If you don't want or need to support Flash and Silverlight clients then you don't need them.

So which one should I pick?

 If you want your API to be adopted in as many places as possible then you should implement all of them! Seriously:

1.       Always make it possible for your API to return JSON.

2.       Support JSONP as much as you can (since you can't use it to upload data that doesn't fit in the URL) and leave it up to the users of your API to assess the security risks

3.       Support CORS by returning the Access-Control-Allow-Origin header on all API responses and implementing the OPTIONS verb as specified in the CORS spec.

4.       If you do the above your API will also support UMP.

5.       Return a crossdomain.xml and clientaccesspolicy.xml file from the top level of your API domain for Flash and Silverlight clients.

Whew!

At Apigee, we have done the above for users of our Apigee Enterprise product, and since it's so flexible we can change what we support as the standards change.

OAuth is improving, but still moving »

We've been following the fast-moving debate in the IETF regarding OAuth 2.0.  OAuth, for those of you who have not encountered it already, is a set of authentication technologies for the Internet designed around the concept of an access token.

Access tokens, in the words of Eran Hammer-Lahav, are like valet keys -- they give the holder access to a specific function, for a specific amount of time. For instance, you might use OAuth to give another web site the ability to read photos from your Flickr profile, but not to modify them. OAuth lets you do this, it lets you go back to Flickr and revoke the web site's permissions at any time, and it does it without requiring that you give the site your Flickr password.
 
The current spec, OAuth 1.0a, is implemented in lots of places, and it solves a lot of problems. However, implementing it is no picnic for either the API provider (the server) or for the developer who builds the client. (There are libraries, of course, not to mention products such as our own that simplify this process.)
 
OAuth 2.0 introduces many changes. The most important is that a client may now use a "bearer token." That's a fancy IETF way of saying that an access token can just be a string that the server gives you. On every request, the client passes that token back to the server, the server checks to see if the token is valid, and you're done. This is much simpler to implement than OAuth 1.0a, but it is only secure if you use SSL for every request. Applications that won't or can't use SSL may still use the old way of transmitting each token, which encrypts the token so that it is safe even if SSL is not used or even if it is intercepted by a proxy like Apigee.
 
However, OAuth 2.0 is far from complete. It is currently undergoing lots of discussion on the IETF mailing list, and the spec draft changes daily.
 
That's why I was surprised to read today that Facebook is using OAuth 2.0 to authenticate its own API. Now, some of the key players in OAuth work at  Facebook, and they have chosen to use only a part of the spec, and the part that's arguably the least complicated. I'm sure that they feel that taking this calculated risk now is in the best interest of Facebook and its developer community, but the possibility remains that the spec will change and Facebook will have to change its implementation to match.

(In fact, at the moment I write this, they do not -- the name of the query parameter that holds the token is "access_token" in the Facebook documentation and "oauth_token" in the latest version of the spec repository.)
 
In the meantime, developers building on top of these APIs may have to contend with OAuth 1.0a (the current spec), OAuth 1.0 (an older version that some sites may still use), the draft form of OAuth 2.0 as implemented by Facebook, and even "WRAP," which introduced some of the concepts used in OAuth 2.0.
 
So the good news is that there are lot of good standards being written that can make it easier to produce and consume powerful and secure APIs. The bad news is that those standards are still changing. So stay tuned, and be careful!

Considerations for ‘open data’ API programs »

In my last post I wrote about how the book digitizing effort is trying to monetize underutilized books online.

For anyone contemplating exposing data or capabilities via APIs to create new revenue streams, there are some important implementation lessons that can be learned.

Have control

In Norway’s Bookshelf project, their free online books can only be read online and only in Norway, and cannot be downloaded or printed out. Similarly, with APIs you need to have a way to control who can access your data and from where. API Identity, API authorization and other API security considerations are a must.

Make it (at least something) free

Google must operate under copyright laws, but it has found a way to freely expose extracts or snippets under ‘fair use’. Similarly, you cannot expect uptake of your content or capabilities if some of it cannot be consumed easily and en gratis. Give at least some of what you offer away for free to spur adoption. Technically, this means being able to control what can be consumed at a granular level.

Get started!

Google, Bookshelf, and Europeana have ideas about how they’ll monetize books they put online, but they are still experimenting with the right business models. Executives shouldn’t wait for an iron-clad business model to present itself… you won’t figure out the right model unless you are ‘in the game’.

Design your API for adoption with ‘true REST’ »

"The only reason you'd have only a SOAP API is because you hate 80% of your addressable market." - @sramji

There's usually little argument that a REST API is easier to use than a SOAP API.

But how important is it to be 'truly' or 'strictly' RESTful? That is, adhering to standard HTTP operations or 'verbs' - GET, PUT, DELETE, POST - on well defined resources, as opposed to the common practice of embedding 'verbs' or operations as methods in a GET URL.

Typically, security is cited as the big advantage of 'true REST' (with some good discussions here and here).

However, a truly RESTful API may help you boost developer adoption. For example, imagine a 'shopping cart' API:

While the above API isn't 'truly RESTful', it's not that hard to use.  But you do have to learn the individual operations and this can get cumbersome if there are a lot of them or as the API evolves.

Instead, this 'true REST API' may be easier to learn and predict as you use more features.

What if we want to list all the shopping carts in the system at any one time? We would add that via an HTTP GET to:

http://api.shopping.com/carts.xml

Query parameters can still serve a purpose - making it possible to specify additional options. For instance, imagine a very large shopping cart, and you want to "paginate" the results. To look at items 20-29, you might use a URL like:

http://api.shopping.com/carts/X.xml?start=20&;count=10

Bonus: True REST makes analytics easier

It's a lot easier to build reports that segregate and analyze traffic by URL than to build logic that tries to do this by methods or combination of methods. Good API analytics helps you optimize features, debug problems, and weed out traffic that can slow down your service.

What if your 'non-strict' REST API is already out there?

It might not be that big a deal if your API is very simple or 'read-only' API with information that isn't too sensitive (such as a free search API). Or you can map a non-RESTful API into a 'truly RESTful" API with custom code or API management tools thatperform API transformations.

* A note on POST VS. PUT

* One way to insert an item in the shopping cart is to use POST to update the shopping cart by sending it a new item. In this case, we are using POST to send the server an instruction that essentially tells it to insert some new content to the existing resource. This is why we use POST -- it is like an "update" in a database.

Alternately, we could add the item by using PUT to a new URL, such as:

PUT http://api.shopping.com/carts/X/items/Y.xml

But if we do this, then we need to somehow give our item some sort of unique URL by picking a value for "Y". This is kind of a strange thing to do, so it may be more natural to use POST and have the response include the URL for the new item, so that we may retrieve it later using GET or delete it using DELETE. Still, sometimes using PUT like this makes sense.

This all comes down to the difference between PUT and POST in the HTTP spec. POST modifies something that already exists, and how that thing is modified is up to the server. PUT replaces the entire contents of the URL with new data. Plus, like GET, HEAD, and DELETE, PUT is idempotent, which means that if you call it more than once, it has the same result every time, whereas POST may keep doing what you ask it to do over and over again

Yes, REST security does exist »

A recent article "Why REST Security Doesn't Exist." postulates, "REST does not have predefined security methods, so developers define their own."

Some good points in here (such as 'don't roll your own')  but I might not completely agree with the premise.

One of the fundamental principles of REST is that it builds on the HTTP protocol -- and the HTTP protocol very much does have "predefined security methods."
 
The basic HTTP protocol supports a way to plug in different security schemes. It also supports OAuth, two-way SSL, and many other mechanisms. Not only does HTTP allow for many security schemes, many of which like HTTP basic are defined by IETF standards, but it also supports a mechanism that allows a server to identify when a request was rejected, if the request was rejected because the security credentials were invalid or because an authorization check failed, and whether the rejection was permanent or temporary. HTTP also includes a mechanism that allows the server to issue a "challenge" that asks the client to re-send a request with a particular type of credentials if it has them. This all adds up to a security method that has proven quite robust over the years and has been extended with new methods such as OAuth when new problems arise.
 
Also, don't forget that different APIs demand different security requirements. An API that offers product catalog information, for instance, with no way to update the information, does not require strong authentication if the owner of the data intends that data to be public anyway. A simple "developer key" that uniquely identifies the sender of the request -- yes, a username "without a password" -- is just fine for that type of API because it is used to identify the user for various tracking purposes, and is not designed to prevent unauthorized users from gaining access to the data.

Cool article and great to see some discussion on this!

New features:  SSL support and “Remember-me” »

Recently we rolled out two new Apigee capabilities.

First, we've added SSL support for APIs that support the HTTPS protocol. Now you can create an Apigee proxy for an API secured by HTTPS—such as the PayPal API—just specify 'https' in the API URL when creating a proxy. You could also change an existing HTTP proxy to an HTTPS proxy. (Of course, the API in question must support HTTPS, else you'll get an error.)

We're also working on adding SSL support for API providers that want to map to their own domain URL (and therefore their own certificate). Stay tuned, and you can review and comment on our proposed SSL Solution at anytime on our community area.   (You will see that we've completed sections 1 and 4 of the proposal to date..)

Secondly, we've added 'remember me' functionality on log-in.  We'll remember your login for 2 weeks so you don't have to specify your password, just head to app.apigee.com to see your proxies. This also means you can bookmark a favorite chart.

Thanks for the continued feedback—we're prioritizing features based directly on your input!

Cloud Security tech talk series: Security and Scalability »

Next in our series of tech talks on cloud security issues, Greg and Ryan Bagnulo, Security Architect for ASPECT-i discuss how scalability can change security requirements and how cloud computing offers new opportunities to fend off attacks on services including. 

• security at high scale - how to preserve the resilency of the busines
• cloud powered security - using elastic cloud resources at the edge to protect core services
• protecting against bot attacks and spikes through security policy enforcement and caching

Check out this talk below, last week's video on PII and Audit compliance ,  and the full series here.

API threat protection pack: 10 XML attack types to guard against »

The cost of IT security breaches has almost doubled from 2008 according to this piece via ComputerWorld Canada.

While we'd love to tell you this is just a problem for our Canadian friends - unfortunately we all need to understand API attack types.

(Remember in our Cloud security tech talks last week we saw that for breaches over a certain size you may even need to issue a press release!)

Here are 10 threats that we cover in our API threat protection policy pack. 

1. Malicious Code Injection:  exploits backend services that use SQL/LDAP/ XPATH/ XQuery statements from user-supplied input.  Servicenet ‘s Malicous Code Injection Detection policy can filter SQL,LDAP, XPATH, XQUERY injection or use Custom Regular Expression, XPATH and XSD technologies to filter the request further.   It also can integrate with anti-virus products to scan for virus in the API requests especially in the attachements or mime contents.
 
2. DOS Attacks: Denial of Service (DoS) intends to prevent an API or Service from serving normal user activity. These malicious attacks includes mega-message and entities attack, recursive element attack, request flooding, larger volume of  invalid requests etc.  The ServiceNet Message Payload protection policy detects various kind of DOS attacks and protect the backend from the attacker.
 
3. Service Information Leakage:  APIs can unintentionally leak information about their configuration, internal workings, or violate privacy through a variety of service errors.  For example verbose and informative error messages may result in data leakage, and the information revealed could be used to formulate the next level of attack. ServiceNet response Message control policies can customize fault/response message reaching the client which can weed out this attack.

4. Broken Authentication, Session id and Keys: Proper authentication, API key and session management is critical to service security. Flaws in this area most frequently involve the failure to authenticate (weak or multiple adhoc authentication schemes), weak session/key tokens that helps attacker to replay or fake the keys or tokens.   ServiceNet’s authentication and API key management policies  provides single point strong authentications and key generation techniques that frees-up API developer from attack risks.

5. Failure to protect API and corresponding Data access: Frequently, authorization is based only on base URI or operation of API.  An attacker can try passing various parameters to this API operation and get access to the data that he not authorized to access.  ServiceNet fle xible authorization policies supports authorization based on various request parameters/data not just URI or Operation name.

6  API Data snooping: Failure to encrypt sensitive API communications means that an attacker who can sniff traffic from the network will be able to access the conversation, including any credentials or sensitive information transmitted. Servicenet’s SSL or XML encryption policies can be used to secure the API data from getting snooped in the communication path.

7. API Request and Response tampering: The API data tampering attack is based on the manipulation of API request and response parameters exchanged between client and services in order to modify application data, such as user credentials and permissions, price and quantity of products, etc. Usually, this information is part of HTTP URI or Header or Body(XML or non-xml).  Servicenet’s SSL or XML signature policies can be used to secure the API request and response message from getting tampered in the communication path.

8. Request Burst: Spikes in API requests might bring down the backend server. Spike Arresting and caching helps the backend services to perform better under various load conditions.

9.  Auditing: If your API is going to be handling money, you may be required by law to adhere to certain security practices and regulations.  One important regulation is auditing every (full or part of) request or response from authorized and unauthorized users.  ServiceNet auditing policy supports very flexible way to log API audit data in various formats to different destinations like Local disk, NFS, Syslog, JMS or Web Services.

10. Threat Detection and Analysis:  Analyzing the threat data is important to find the failures and fix those failures on the API infrastructure. ServiceNet’s analytics policy provides capability to visualize and analyze various API errors or failures. It can also provide various patterns or rates of these failures that help an architect or developer to fix the problem in his or her API.

For more on API security and threat protection, check out our compliation of API roadmap issues - Is your API Naked?  And let us know if you like to see the demo of this policy pack in action.

(Senthil Doraiswamy is a product manager at Sonoa Systems.)

(And thanks misocrazy to for the photo..)

Cloud Security series - issues around PII, privacy, and audit compliance »

Greg recently sat down with Ryan Bagnulo, Security Architect for ASPECT-i, to discuss a number of cloud security concerns and issues.   

We captured these discussions in six short videos, each focusing on a  topic.  Here are the first two on PII, data filtering, and audit and regulatory concerns,  (see the full series here.)

In this first video, Greg and Ryan set things up with discussions on:

• Challenges in deploying cloud, starting with: should you trust your cloud administrator?
• Good data for early cloud adoption (such as public data like news, stocks)

This 2nd short focuses on:

• issues around PII (personally identifiable information)
• counter-measures, such as de-identifying data with filtering, screening or access control
• privacy and regulatory risks around stored in the cloud.
• best practices for protecting data
• implications for violating security breaches privacy regulations

We'd love your thoughts and comments.

Protect your API: Twitter’s Denial of Service Attack and API Security »

Lots of news on the Twitter attack last week. There’s general consensus that it was a distributed denial-of-service attack (DDOS) that it targeted a particular account.

DDOS attacks are tricky things. There’s no one technique, product, or protocol that will stop them. That’s what makes them so nasty. To defend against one, a company needs to be able to quickly take countermeasures at all the different levels of the protocol stack, including firewalls, routers, load balancers, and even in the application itself. And even more importantly, you need to have experienced, well-trained operations people who know how to quickly identify an attack and come up with a way to minimize its effects.

When a DDOS attack affects a web service – and there’s an good chance that the Twitter attack made use of the Twitter API – then proper traffic management at the API level is an important part of this stack of protection.

For instance, if a DDOS attack targets a particular user account, or API key, it may be possible to block those particular requests before they get to the back-end application servers, so that the effect of the attack is limited. Or, if the attack always sends some of the same parameters in an HTTP query string, or in the body of an HTTP POST, it may be possible to detect those patterns and reject the requests, again before they reach the back-end application servers. In the worst case, it may even be necessary to shut off access to a particular web service or web service operation, allowing other services to function as best they can while the attack is neutralized by other means.

In other words, just as it may be possible to stop a DDOS attack by blocking a range of IP addresses, or redirecting certain URLs, or configuring traffic shaping in a router or an ADC, it may sometimes be necessary to stop a DDOS attack closer to the application level. The ability to inspect requests at the web services level and take action based on the request content gives an operations team one more weapon to use against an attack.

The Twitter API already has a usage quota on each user account, a rate limit on each IP address, and extensive caching. Imagine how much easier a DDOS attack might have been if they didn’t even have those things.

-Greg

(Greg Brail is the CTO of Sonoa and writes on API Security topics such as API Keys, OAuth, and API Security Recommendations.) 

Tech Talk: API Security and Threat Protection »

Greg recorded a few whiteboard talks last month - this one is a good summary of recent posts on API Keys, API security recommendations, and OAuth best practices. 

(And here are some of our high-level API security features if you are looking into this.)

OAuth — Take care with those keys! »

A lot has been happening with OAuth recently. Earlier this year a security hole was discovered in the protocol which exposed it to a potential “social engineering” attacks.  However, the OAuth community is working on a revision to the spec that will eliminate this particular hole.

Last week we wrote a bit on OAuth as an option for API security.  But today I wanted to bring up a related OAuth issue - how do you securely manage all those keys?

With traditional username / password authentication, good security practices require you don't just have a big database on the back end with a list of unencrypted passwords. Instead, a hash of the password is stored, preferably using a salt. So someone who can read the password file can verify they have the right password, but cannot see the actual password.

It is still critical to protect access to these encrypted passwords. Otherwise, an attacker can mount a dictionary attack to try and crack them. However, even if someone gains access to your entire database of encrypted passwords, they can still only easily gain access to lousy passwords. At least users who choose secure passwords are relatively safe. (It is also critical to protect access to the cleartext password, but at least this mechanism doesn’t require that it be stored in a database for all to see.)

As networking and middleware people, we spend a lot of time thinking about the security of our network protocols, and especially ensuring that someone eavesdropping on a network cannot grab our passwords and other sensitive data as they fly by. But how many times have we heard of a security breach caused by a stolen laptop? I would argue that protecting so-called “data at rest” is just as important, or maybe even more important, as protecting the data flying around your laptop.

Now, back to OAuth. Each “user” in OAuth holds something called an “access token,” which is like a username, and a “token secret,” which is like a password. When a request is sent over the network containing an OAuth authentication token, a bunch of data in the token is encrypted using the token secret, but the secret itself is never sent over the network. That way, regardless of whether SSL is in use, there is no way to gain access to the token secret by sniffing the network.

However, on the server side, in order to validate the OAuth token, the server must make the same calculation that the client made when it encrypted the data to put in the token. That means that both the client side and the server side in OAuth must be able to read the unencrypted token secret from some sort of database. Without it, OAuth doesn’t work. There’s no set of standard ways for storing those keys like there are for passwords, so presumably different implementations are storing them in different ways.

As a result, any client and any server that uses OAuth has to take extra-special care with all those token secrets. Otherwise, anyone who gets access to the database of tokens and secrets used by the back end servers immediately has access to all the OAuth-enabled accounts.

I am not suggesting a change to the OAuth protocol here — it solves an important problem. However, I am suggesting that anyone who implements either the “service provider” or “consumer” side of OAuth take very special care of those tokens!

For instance:

•     If they’re on a regular disk file, protect them using filesystem permissions, make sure that they’re encrypted, and hide the password well.
•     If they’re in a database, encrypt the fields, store the key well, and protect access to the database itself carefully.
•     If they’re in LDAP, do the same.

Come to think of it, perhaps the world needs a standard LDAP schema for storing OAuth secrets in a secure way. Anyone care to make a proposal?

Don’t roll your own: API Security Recommendations »

Let's boil up the examples and common pitfalls from our last two entries on API Identity and Authorization and more API security choices.   

Use API Keys for non-sensitive data (only): 

If you have an “open” API - one that exposes data you’d make public on the Internet anyway -  consider issuing non-sensitive API keys. These are easy to manipulate and still give you a way to identify users. Armed with an API key, you have the option of establishing a quota for your API, or at least monitoring usage by user. Without one, all you have to go on is an IP address, and those are a lot less reliable than you might think. (Why don’t web sites issue “web site keys?” Because they have cookies.)

For example, the Yahoo Maps Geocoding API issues API keys so it can track its users and establish a quota, but the data that it returns is not sensitive so it’s not critical to keep the key secret.

Use username / password authentication for APIs based on web sites: 

If your API is based on a web site, support username/password authentication. That way, users can access your site using the API the same way that they access it using their web browser. For example, the Twitter API supports username/password authentication, so when you access it using a Twitter API client like Spaz or TweetDeck you simply enter the same password you use when you use the twitter.com web site.

However, if you do this, you may want to avoid session-based authentication, especially if you want people to be able to write API clients in lots of environments. It is very simple to use “curl,” for instance, to grab some data from the Twitter API because it uses HTTP Basic authentication. It is a lot more complex if I instead have to call “login,” and extract a session ID from some cookie or header, and then pass that to the real API call I want to make...

OAuth for server-to-server APIs that are based on web sites:

If your API is based on a web site (so you already have a username / password for each account) and the API will also be used by other sites, then support OAuth in addition to username / password authentication. This gives users a lot of peace of mind, because they can authorize another site to access their sensitive data without giving that site their password permanently.

Use SSL for everything sensitive:

Unless your API has only open and non-sensitive data, support SSL, and consider even enforcing it (that is, redirect any API traffic on the non-SSL port to the SSL port). It makes other authentication schemes more secure, and keeps your user’s private API data from prying eyes – and it’s not all that hard to do.

Don’t roll your own!

If the above suggestions still don’t apply to you, then keep looking – between OAuth, OpenID, SAML, HTTP authentication, and WS-Security, there are a lot of authentication schemes, and each has its pros and cons.

So wrapping up API security in our series on 10 API roadmap considerations. Here are some suggested questions you may want to ask when putting together your API security roadmap:

How valuable is the data exposed by your API?

• If it’s not that valuable, or if you plan to make it as open as possible, is an API key enough to uniquely identify users?
• If it is valuable, can you reuse username and password scheme for each user?
• Are you using SSL? Many authentication schemes are vulnerable without it.

What other schemes and websites will your API and users want to integrate with?

• If your API will be called programmatically by other APIs, or if your API is linked to another web site that requires authentication, have you considered OAuth?
• If you have username/password authentication, have you considered OpenID?
• Can you make authorization decisions in a central place?

What other expectations might your customers have?

• If your customers are enterprise customers, would they feel better about SAML or X.509 certificates?
• Can you change or support more than one your authentication approach for diverse enterprise customers?
• Do you have an existing internal security infrastructure that you need your API to interact with?

Up next:  API Data Protection and thanks to Torbin H. for the photo.

More API Security Choices - OAuth, SSL, SAML, and rolling your own »

(This entry continues our API Roadmap consideration series and is a continuation of last week's API Identity and Authorization.

Session based Authentication – cumbersome with RESTful APIs

Lots of APIs  support session-based authentication. In these schemes, the user first has to call a “login” method which takes the username and password as input and returns a unique session key. The user must include the session key in each request, and call “logout” when they are done. This way, authentication is kept to a single pair of API calls and everything else is based on the session.
 
This type of authentication maps very naturally to a web site, because users are used to “logging in” when they start working with a particular site and “logging out” when they are done.
 
However, session-based authentication is much more complex when associated with an API. It requires that the API client keep track of state, and depending on the type of client that can be anything from painful to impossible. Session-based authentication, among other things, makes your API less “RESTful” - an API client can’t just make one HTTP request, but now a minimum of three.
 
The Role of OAuth

Today, many APIs also support OAuth authentication. OAuth was designed to solve the application-to-application security problem. The idea of OAuth is that it gives the user of a service or web site a way to conditionally grant access to another application. OAuth makes it possible for a human user to individually grant other APIs or sites access to their account without sharing their actual password. It works by giving a “token” to each API or site that will access the account, which the user may revoke at any time they wish.

For instance, if web site FooBar.com wants access to the Twitter API on behalf of John Smith, then OAuth specifies the protocol that FooBar.com must go through to get an OAuth token for the Twitter API. Part of this process requires John Smith to log in to his Twitter account using his normal username and password (which, in the OAuth protocol, are never seen by FooBar.com) and authorize FooBar to access his account. The result is that FooBar.com will have an OAuth token that gives it access to John Smith’s account. If, later on, John Smith decides he no longer trusts FooBar.com, he has the option to revoke that OAuth token without affecting his regular password or any other accounts.

This process makes OAuth the ideal for communication from one application to another – for instance, allowing MySpace to post photos to your Twitter account without requiring that you enter your Twitter password every time you want to do it – but it can be used for any kind of API communications as well.
 
Two-Way SSL, X.509, SAML, WS-Security...

Once we leave the world of “plain HTTP” we encounter many other ways of authentication, from SAML, X.509 certificates and two-way SSL, which are based on secure distribution of public keys to individual clients, to the various WS-Security specifications, which build upon SOAP to provide... well, just about everything.

An API that will primarily be used by “enterprise” customers – that is, big IT organizations – might consider these other authentication mechanisms such as an. X.509 certificate or SAML for more assurance over authentication process than a simple username/password. Also, a large enterprise may have an easier time accessing an API written to the SOAP standard because those tools can import a WSDL file and generate an API client in a few minutes.

The idea is to know your audience. If your audience is a fan of SOAP and WSDL, then consider some of the more SOAP-y authentication mechanisms like SAML and the various WS-Security specifications. (And if you have no idea what I’m talking about, then your audience probably isn’t in that category!)

 
Rolling Your Own

In between OAuth, HTTP Basic, and the basic API key are many alternatives. It seems that there are as many other API authentication schemes as there are APIs. Amazon Web Services, Facebook, and some Google APIs, for instance, are some big APIs that combine an API key with both public and secret data, usually through some sort of encryption code, to generate a secure token for each request.

The issue – every new authentication scheme requires API clients to implement it. On the other hand, OAuth and HTTP Basic authentication are already supported by many tools. The big guys may be able to get away with defining their own authentication standards but it’s tough for every API to do things its own way.
 
SSL

Most authentication parameters are useless, or even dangerous, without SSL. For instance, in “HTTP Basic” authentication the API must be able to see the password the client used, so the password is encoded – but not encrypted – in a format called “base 64.” It’s easy turn this back into the real password. The antidote to this is to use SSL to encrypt everything sent between client and server. This is usually easy to do and does not add as much of a performance penalty as people often think. (With Apigee’s products, we see a 10-15 percent drop in throughput when we add SSL encryption.)

Some security schemes, such as OAuth, are designed to be resistant to eavesdropping. For instance, OAuth includes a “nonce” on each request, so that even if a client intercepts an OAuth token on one request, they cannot use it on the next. Still, there is likely other sensitive information in the request as well.

In general, if your API does not use SSL it also potentially exposes everything that your API does.

Next time, we'll make some recommendations among different options - or get our full paper here.

TrueCredit.com API case study »

Scott Metzger, CTO of TrueCredit.com was kind enough to take some time to talk about their Consumer Connect API program and some of the technical challenges that they have addressed using Apigee's API Gateway.

Scott wanted to make life easier on his development team as they ramped up their number of APIs, partners and traffic volumes.  Here, he describes how he uses the technology as a 'policy layer' to provide API analytics, fine-grained data protection, and caching in an API Gateway. In this case, Apigee Enterprise is deployed on-premise virtualized software.

We're very excited to be working with Scott and TrueCredit, and check out the full TrueCredit Case study. 

Do you need API keys?  API Identity vs. Authorization »

(This is part 3 in our series on "Is your API naked? 10 API Roadmap considerations")

We’ve seen very few API providers with a completely open API – almost all employ at least one of these:

• Identity - who is making an API request?
• Authentication - are they really are who they say they are?
• Authorization – are they allowed to do what they are trying to do?

Do you need them all?  Maybe not.  Some APIs only need only to establish identity and don’t really need to authenticate or authorize.

API Identity vs. Authentication - Compare Google Maps and Twitter  

Take Yahoo and Google maps – they are fairly open.  They want to know who you are but they aren’t concerned what address you are looking up. So they use an “API key” to establish identity, but don’t authenticate or authorize. So if you use someone else’s API key, it’s not good but not a serious security breach.

The API key lets them identify (most likely) who is making a API call so they can limit on the number of requests you can make. Identity is important here to keep service volume under control.

Then take Twitter’s API -  open for looking up public information about a user, but other operations require authentication. So Twitter supports both username/password authentication as well as OAuth. Twitter also has authorization checks in its code, so that you cannot “tweet” on behalf of another user without either their password or an OAuth key to their account. This is an example of an API that implements identify, authentication and authorization.

The “API Key” – do you need one? 

API keys originated with the first public web services, like Yahoo and Google APIs.  The developers wanted to have some way to establish identity without having the complexity of actually authenticating users with a password, so they came up with the “API key,” which is often a UUID or unique string. If the API key doesn’t grant access to very sensitive data, it might not be critical to keep secret, so this use of the API key is easy for the consumers of the API to use however they invoke the API.

An API key gives the API provider a way to (most of the time) know the identity of each caller to maintain a log and establish quotas by user (see the last section). 

Usernames and Passwords – again, see Twitter

With more sensitive data a simple, API key is not enough, unless you take measures to ensure users keep the key secret. An alternative is username/password authentication, like the authentication scheme supported by the vast majority of secure web sites. 

It’s easiest to use “HTTP Basic” authentication that most websites use. The advantage of using this technology is that nearly all clients and servers support it. There is no special processing required, as long as the caller takes reasonable precautions to keep the password secret.

Twitter simplifies things for their users by using usernames and passwords for API authentication.  Every time a user starts a Twitter client, it either prompts for the username and password to use, or it fetches them from the disk (where it is somehow scrambled or encrypted where possible). So here it makes a lot of sense to have the same username / password for the Twitter API that it used for the web site.

Usernames and passwords also work well for application-to-application communications. The trick - the password must be stored securely, and if it’s being used by a server, where do you store it?  If you are running an application server that uses a database, you already have solved this same problem, because the database usually requires a password too. Better application server platforms include a “credential mapper” that can be used to store such passwords relatively securely.

There is a lot we'd like to write about around security so we'll split this up into a couple entries.  Next time:  Session-based authentication, OAuth, SSL and WS-Security, and rolling your own.

So you want to open an API? »

Greg Brail, our CTO, took advantage of a break from O'Reilly Velocity and a new Flip HD ultra to record a series of four short whiteboard talks on issues many face when opening an API - visibility, security, traffic management, and more.  Here is the first clip and you can preview all of these and more API case studies on our Apigee youtube channel.).  

We love feedback and these are quick to do - so any more topics you'd like to see, please let us know..

Challenges when building APIs »

If you’re planning to build an API and expose it to the Internet then you’re going to have to face some challenges that you won’t necessarily find when building an internal web service. For instance:

Design. The best APIs are the simplest, but designing a simple API isn’t easy. Plus, what’s simple to one user population isn’t simple to another. A “REST-style” API like Twitter’s is great for AIR programmers or Perl hackers but someone accessing it from inside a big web app server stack might actually find it easier to use a SOAP web service with a WSDL. On the other hand, a SOAP-only API would have been death for Twitter because it would have meant that those tens of thousands of Perl hackers would have had a heck of a time using it in the first place.

Compatibility. Let’s say you don’t get the design right the first time — and how often does that happen? How many “old” versions of your API can you afford to keep running to keep clients functioning? Are are you willing to tell your users, “sorry, we changed the API and now you have to re-write your apps.”

Authentication and Authorization. What does your API do? If it just lets you look up public information, maybe you don’t need authentication. But are you planning on using it with more sensitive data? Will people be using your API to spend money? They’re going to expect that they have to authenticate using a username and password at the very least. There are quite a few ways to do that — which one(s) will you choose? How will you manage all those accounts?

Threat protection. Is there a possibility that a malformed API request can cause your servers to go off in la-la-land, trying to execute an impossible query? Did you code everything write to prevent a SQL injection attach? What if a client sends your servers some bizarre XML — will they run out of memory or crash?

Latency. Since the goal of your API is to provide a service over the Internet, then you will have to live with anywhere up to several hundred milliseconds of latency just to get to and from your API. If each API request takes hundreds more milliseconds, or even several seconds, to run, then how will that affect the perception of your service?

Visibility. Who is using your API? How often? How do the patterns change over time? What kind of latency are they seeing? How many errors do they get? Do different users see a higher error rate? Is the user you signed up last week actually using the API? These are all questions you will want to answer in order to serve your customers better.

Rate limiting. How do you plan to limit user access to your API? Sometimes the right answer is to do nothing — and this is often the right answer for an internal system, where saying “no” is not an option. But for a public, Internet-based API, you owe it to yourself to at least protect your API against disaster — a user who decides today’s a great day to see if they can call your API 100 or 1000 times per second, or one that makes a programming mistake and codes up an infinite loop, or worse. And if you’re planning on a larger user population, then a formal set of quotas makes a lot of sense, which is why Twitter, Yahoo, Google, Amazon, and others all put limits on how much you’re allowed to use their APIs before you give them a call and let them know what you’re up to.

Next, I hope to dive into what we're seeing for each of these and more in detail -Greg