This is the third continuous year that I have had the good fortune of attending the Gluecon conference. Held May 21-22 in Broomfield, Colorado, the conference focuses on big data, cloud, and API issues. Admist the technical chat, a number of the sessions that I attended also had an emphasis in enhancing software development culture. Below is an abridged version of my notes.
Social Glue
Rachel Chalmers, (ignition parterns)
Through many natural examples, ideal team size seems to be 2-6 people.
Breaking up the Problem
- solving large problems is very heard
- breaking down the problem is itself is a problem
- but it can be done
- good interfaces make good solutions
- better interfaces are cultural as well as technical
Reference to a Velocity 2009 ppt, called "10 deploys per day - dev and ops at Flickr". It proposed a new social contract partially in software but also 'codified' in a culture. The code was the stored wisdom of the ops team. They set ground rules of how people would interact with the software and among themselves. They cultivated a healthy attitude toward each other when the inevitable disasters would happen. This trust in the team reduces cognitive load.
"Enterprises want to move from disaster recovery to disaster indifference." Chris Stevens
A company is not just a technology or a legal structure. A company is a group of companions. Is is a set of people that work better together than alone. Rachel also cited a Netflix culture document, "Seven Aspects of our Culture". A culture like Netflix's is a key differentiator. Rachel is willing to bet her career that an open, transparent organization can out compete a coercive, controlling one. Netflix is highly aligned, loosely coupled.
Code is stored wisdom. Culture is about creating a safe and supportive environment where the best ideas can be captured into code.
CIOs want to build service-orientated architecture. The move from waterfall to agile is a response to a need to more quickly in the face of competitive pressure. It illustrates how every company should think of itself in the software and information business.
Private IT will never compete with the public cloud. And "not developed here" approaches can't compete with SaaS. Desktop gives way to mobile. But mobile is training wheels compared to the coming IoT (Internet of Things). Silos give way to APIs and services. These process are all reflection of the need to better capture collective intelligence for competitive advantage.
Code is stored wisdom. Culture allows wisdom to surface from groups. Clean interfaces enable groups to speak with one voice, reducing communication overheads.
Containers All the Way Down: A Blueprint for Upgrading the Internet
Solomon Hykes, Docker
Docker is an open source project that standardizes how a project is packaged and subsequently executed in a sandboxed environment. The pieces are detached from any specific machine.
The challenge we all have is that the applications that we're now building don't neatly fit notions of platforms of the past. We are able to point at individual products but things like cloud services are fuzzy concepts; it doesn't capture the entirety of what we're doing. We cobble together tools that are not fit for working together. Creating the glue to make these disparate pieces work takes a lot of time and effort.
"apps we are building are made of services: discrete logical loosely couple composable components"
Composability and remixability are important. They need to talk with each other over network messages. They also need to be independent of the machine they're running on. Hardware dies, becomes obsolete, and therefore the software - with expectations of 100% uptime - needs to be independent of infrastructure.
"My thought in the 70s was that the Internet we were all working on alongside personal computing was a really good scalable design, and that we should make a virtual internet of virtual machines that could be cached by the hardware machines. It's really too bad that this didn't happen." - Alan Kay (creator of OOP, and Smalltalk)
All Docker does is try to gain consensus around a key thing is missing: regardless of how people wish to implement to solution there is a uniform place for people to go build stuff.
The new problems:
- clustering
- orchestration
- discovery
- identity and trust
These problems have been solved. We just need to standardize the interfaces between them.
"Docker is a standards body that anyone can join just by writing code."
The Parallel Universes of DevOps and Cloud Developers
Donnie Berkholz, Redmonk
In his job, Donnie has gained an expertise of quantifying things like open source communities. When observing cloud based solutions, becomes apparently that developers are put in the position of being their own sysadmins. DevOps - the traditional division with sysadmin functions - are not aware of what the developers are doing.
It's not about tools, it's about people and culture.
Part of the separation problem is because of differences in culture. In order to facilitate faster deployments, we need to tear down the silos around the separate groups. Silos are difficult to maintain when things are entirely open.
"Completely disagree that the solution for devs and ops working together is culture. Culture reflects norms, not causes them" Bernard Golden
"Infrastructure as code" are projects like CFEngine, Puppet, Chef, Saltstack. It's not a new idea (CFEngine came out in 1993). However, the tools came to the forefront with the need to matriculate deployments to the cloud.
The best Dev/Ops Experience wins. Common barriers to a great DevOp experience include:
- inability to find the download
- requiring registration
- tricky, multi-step installation and/or configuration
- having a poor first dev experience
- seeing no community where I can turn to for support
Optimizing for cloud deployment is very different than just building software that can be run in the cloud. For Ops its not about the run-time experience, it's about the build time experience.
In dev world, people choose tools they enjoy, the same as with developers. For example, an obscene amount of developers choose mongoDB because of how delightful the development experience is. Listed in order of popularity, developers choose:
- mongoDB
- redis
- mysql
- mongolab
But Ops prefer the tools that best support them:
- postgreSQL
- Cassandra
- riak
Today we have two separate communities, living in parallel. They do similar things but rarely does either group reach out to each other. What is the common ground? Continuous delivery and DevOps are extremely similar things, but continuous delivery is thought of as a code thing. DevOps is thought of as babysitting servers. That needs to change.
Deconstructing Women in Technology
Ingrid Alongi, Quickleft
"More girls tend to associate computing with 'typing,' 'math,' and 'boredom,' while boys are more likely to associate computing with 'video games,' 'design,' 'electronics,' 'solving problems,' and 'interesting'" - 2008 ACM Study
There is a problem in the discussion on women in technology; the problem is that we're not focused on the right problem. It is the equivalent of "Surviovorship Bias", a phenomenon observed during WWII. At that time the Allies wanted to increase the odds that bombers would successfully return from their missions. Armoring the entire plane was impractical; it would make them too heavy. So they examined where on the planes they found bullet holes. The intention was that they would armor these spots.
While it was a nice sentiment, it was solving the wrong problems. Instead of studying where the bullet holes were on planes that made it back to base, they should have studied where holes appeared in planes that failed to return. Those were the locations that needed additional armor.
A similar problem exists when discussing women in technology. We're looking in the wrong place. It's not a matter of convincing women that technology is cool. It's a problem of culture; women constantly feel pressure to prove their code is just as good as a man's. Or woman feel singled out. They feel conspicuous. They are never made to feel that they are just another member of the team. So they vote with their feet, and leave the industry.
Setting out to explicitly "build a more diverse team" runs the risk of being misconstrued as a touchy-feely effort. However, committing to building a high-performance team not only clearly aligns with the business goals, but will naturally require diverse inputs. Seven ways to promote a better performing team:
- Make a commitment to seeking out original, diverse thinkers and providing a trusting, safe environment where ideas can be thoughtfully heard, considered.
- Host team-building events during office hours. If the only way to talk to boss is after hours at the bar, then that's a turnoff to some types of people.
- Provide multiple outlets for people to reach out to leadership. Empower people to freely talk to those with the power to change things.
- Examine hiring practices. References to "design guy" in long term strategy will likely elbow female candidates from considering themselves for position. Likewise, beware of loaded words or job descriptions that appeal to one group over another.
- Examine your bonus and raise criteria. While you may hire good engineers only a certain subset may be good negotiators. If left unchecked, this may be perceived as wage discrimination.
- ?
- A job is a professional place of work, but that doesn't mean it can't be fun.
API SDKs will Ruin Your Life
John Sheehan, RunScope
Why do SDKs have such an uptake among providers? Because SDKs are on the critical path to developer success for the APIs.
The problem is that to support SDKs in the 6 major languages (.NET, Java, Ruby, Python, PHP, node), and each version has multiple versions. The result is a tremendous amount of developer pain. And then GoLang comes along.
Can autogeneration be a solution? Not really. That is a great way to create mediocre code.
Can it be left to the dev community? Perhaps. But that also has hit-or-miss results and can reflect badly on the service because of bad initial experience.
Devs love SDKs. They are used because:
- Enable Rapid Prototyping
- Fill Gaps When Language Support is Deficient
- Negotiate Complex APIs (or bad APIs, but if a bad API requires an SDK to be usable, shouldn't the API be rewritten?)
- Facilitate Native API Usage
Versioning of the SDK in conjunction with the API is frustrating. Adds headaches to deployment. Other SDK dangers include:
- Using more than one (because of library conflicts)
- Community-built (updates are out of your control)
- Introduces many dependencies
- May become an inactive project
DevOps vs the Enterprise: What We can learn from Mainframe (Going Big and Screwing Up on DevOps Implementations)
Mike Baukes, ScriptRock
Quite often, the business doesn't understand what you're doing in DevOps.
With agile, things can work really well when the familiarity and ability of each member is the same. When there is a high degree of variability then things can become difficult; people get overwhelmed.
There are challenges to devops. Things standing in the way of even deploying simple updates:
- teams spread across geography and org charts
- average request fulfillment was 2 weeks
- riddled with process debt
In building the team culture they recruited the best, but that immediately caused tension and elitism and arrogance began to grow. While they had done a great job selling an approach from the C-level down they hit wall with those in the trenches.
Technical debt was a huge barrier. Spent 90% of time writing code and 10% on education. When it comes down to it, it was faster to code it themselves. Didn't invest enough in the making a good solution. Entire 8 week project was blown away six months later because not enough time was spent on education.
The lessons learned during the implementation:
- spend time upfront being empathetic with the company you're coming into; there's a reason why things have evolved the way they are
- don't automate what you don't understand
- DevOps is an organizational change
- software in production and but not in dev is #fail (leads to a shifting sands of patch management, reporting, etc if otherwise)
How would Mike do things differently today?
Wave 1
- know what you have
- configuration drift detection
- configuration visibility across all devices
Wave 2
- map accountability to org
- execution doc
- pre-flight checks
Wave 3
- test driven automation
- service driven infrastructure
- eliminate duplicate documentation
"Don't automate what you don't understand" And don't automate what isn't working, either. Omar Sultan
API Track Introduction
Alex Williams, The New Stack
It used to be about getting the best efficiencies out of a stack. Now it is about the speed with which you can bring new products to market; whether or not it is efficient doesn't matter. As long as the first attempt is efficient enough for the consumer, that wins.
"We have seen the emergence of lightweight apps... but now we are starting to see the emergence of the lightweight infrastructure."
-
"The API is the single point of truth. We saw how effective that was for supporting apps. That's why we're starting to see the emergence of APIs in the infrastructure environment."
The "single point of truth" is threatened, however, by the Oracle vs Google ruling. If everyone is forced to create custom interfaces for fear of being sued, we run the risk of destroying the speed of interoperability and deployment, the original promise of APIs.
It's no longer a technology matter. An understanding of the courts and broader culture sentiment is increasingly going to matter.
API First: Designing Your Product from the Platform Out
Kirsten Hunter, 3Scale
The idea of API First is that you build the API first before anything else; including websites that traditionally would be server-side code outputting HTML. It involves loving your API and making it a first class product. You want the API to drive an experience that is meaningful to the end user.
APIs that are organically grown resemble an old potato that has left to sprout. Ad-hoc API development leads to duplicate code, features without specific use cases, and inconsistent interfaces. APIs that are perceived as "another thing" to do (in addition to feature development, tests, etc) will never scale or be loved.
The API First Development model is the MVC strategy taken to the next level. It is a uniform contract providing consistently across view clients (mobile, web, other).
Examples of API first companies:
- LOB (cloud printing)
- Instagram (started mobile first, but used mobile API to power a website)
- 3Scale
- Etsy (with v3, even website is on the API)
Kirsten doesn't encourage the use of SDKs. Anything that provides a level of abstraction from the API will provide levels of difficulty in understanding and maintainability (not to mention providing another place for bugs to hide). If an API is so difficult to use, fix that first rather than building an abstraction on top of it.
"If you want your API to remain relevant make it the central interface to your biz logic for both product and integrations" - James Andrew Vaughn
Why Your Next API Should Be Designed By a Linguist
Rebecca Standig, Keen IO
What is linguistics? Linguists look for patterns. In relation to APIs, morphology and phonology are pertinent. They think about syntax in very abstract, meta ways. The same processes are important when developing APIs. Certain language structures are difficult to pick out given the limited amount of information, something not unlike working with customers.
When developing the language of an API, consider:
- do you have a grammar
- that makes sense
- that can be intuitively guessed at by those familiar in your field but not with your implementation
- and be broken down into granular, component parts
- with a syntax (the implementation details) abstracted away from the user?
Common linguistic phenomenon is that complex aspects becomes increasingly less complex over time. This holds true for APIs as well. If something is too difficult to accomplish (for example, requiring multiple calls to compose a single interface) it will be refactored over time.
Security Testing with APIs (Security Scans)
Michael Giller, SOAPUI
First type of security vulnerability are injections:
- sql injection
- xpath injection
- code injection (watch out for eval() func)
- log injection (people can soak up cycles chasing planted ghosts)
- XML External Entity Injection
Cross Site Scripting (XSS); allows people to bypass same origin policy.
"Sensitive information exposure test is brilliant. Seeing service versions, stack traces -- turns my stomach." Neil Mansilla
Denial of Service attack - be able to recognize when to recognize it is under attack. Within SoapUI, using load testing to verify the capability of a server to handle exponential traffic surges.
Check user permissions. Make sure that your users can only access the information they need to access.
Watch out for sequential IDs - they are just begging for someone curious enough to act on "what if" questions.
"In addition to security, it also protects from exposing business growth metrics." Neil Mansilla
Fuzzing Scan generates random input for specified request parameters for a specified number of requests. Helps expose whether you're returning hints for attack when the API is unsure what to do with what it is given.
How to Secure Your REST API the Right Way
Les Hazelwood, Stormpath
Best Practices
API Keys, not Passwords
- Entropy (a key is larger, random than a human-friendly password)
- Independence
- Speed (if using keys, don't necessary have to do processor intensive algorithmic hoop jumping)
- Reduced Exposure
- Traceability
- Rotation
- Authenticate every request (it's supposed to be stateless; sessions are discouraged)
- Encrypt API key secret values (permanent data store protection)
- Avoid Sessions (no XSRF [cross-site forgery] if there is no session)
Identifiers
- should be opaque
- secure random or random/time UUID
- URL-friendly 'base62' encoding
- avoid sequential numbers:
- distribute ID generation load
- mitigate fusking attacks
Query Injection
Always sanitize and validate every input.
Redirects and Forwards
This is where clients and/or customers can declare their own URLs to be sent to after the server processes things. * Avoid if possible * if used, validate the value and ensure authorized for the current user
TLS
- Use TLS for everything
- Once electing to TLS:
- never redirect
- never switch back and forth
- Cookies: set the 'secure' and 'httpOnly' flags for secure cookies
- backend infrastructure connections use TLS too
- configure your SSL provider to support only strong (FIPS 140-2 compliant) algorithms
- use cipher suites w/ perfect forward secrecy
- keep your TLS certificates valid
- app-level encryption + TLKS for most secure results
Configuration
- do security testing as part of continuous integration suite
- keep security patches up to date
- use same config in Dev, Prod, QA (docker is great for this)
- externalize passwords/credentials
Storage
- sensitive data encrypted at rest
- encrypt offsite backups
- strong algorithms/standards
- strong encryption keys and key mgt
- strong password hashing
- external key storage
- encrypted file system
What Makes a Great Developer Company?
Ethan Kurzweil, Bessemer Venture Partners
"Making the Case for the Developer as the Center of the Universe"
Why developers? Why make companies that sell to them?
We now have examples of successfully developer-lead companies (Facebook, Yahoo, Google). Microsoft has greatly benefited from catering to the developer ecosystem. Even with the recent negative press coverage, Microsoft still has 2x the free cash flow than Google.
Because they are more central to competitive advantage, developers are now making spending decisions. References the O'Reilly book, [The New Kingmakers, and how the CIO is last to know. Developers are bringing in their own tooling to accomplish tasks; platform decision making is driven even more bottom-up than top-down.]
eBay paid $800 million to buy Braintree because they had a beloved developer platform, while PayPal was not. Braintree is now replacing the existing PayPal infrastructure.
The business models are strong.
Cost of Acquisition
- low hurdle to adoption
- organic usage drives business
- marketing covers sales and support initially
social media is accelerant (devs talk a lot!)
Lifetime Value of Customers
very sticky and hard to rip out
- enjoy economies of scale
- net negative churn; as customer's usage grow, so does your revenue
Examples of companies doing it right:
- sendgrid
- github
- twilio
- new relic
What makes a great developer company?
The seven commandments of building the biggest company possible (not all need to be met, but getting most of these help):
- delivered as a service that can be metered (as opposed to on premise where you lose touch with usage patterns) - ex. twilio
- Your relationship grows with their business (economics of dropbox on amazon); need to capture value of the business growth you're facilitating
- Replaces something that companies already pay for (Amazon Web Services replaces the pain and maintenance of keeping services yourself. Easily allows company to find budget to direct to you
- Offers an amazing user experience (as good as a consumer product). Just because it's a dev company doesn't mean that devs deserve bad experiences. (Mashape, Stripe)
- Devs love it and aren't afraid to express their love (what do you get when you do a twitter search for a product?)
- Exhibits strong network effects (does it get stronger as more people use it, like Stackoverflow - the more people answer questions, the more valuable the service is)
- Eliminates the need for non-core skillset that no one enjoys (Stripe, Stormpath, Oauth provider)
- (bonus!) Democratizes Development - does it extend the power of software and engineering to more people beyond traditional developers? (Optimizely, ability to run an A/B test). Success is measurable through metrics. Takes 18 million programmers (worldwide) and opens it up to the 400 million IT workers.
Carpe Datum! Time for Software Psychologists to Become Analytics Heroes
Bjorn Freeman-Benson, New Relic
Bjorn realized that much of what we do in software is about listening to other people. That is why he lists "software psychologist" as his job title on LinkedIn.
It is one thing to have bugs in the program. But, as he learned the hard way while developing UrbanSim, there may be bugs in the larger mental model. The software may be behaving perfectly, but if the model is incorrect the results will still result in the same disappointment.
Psychologists use an assortment of evidence-based treatments to help people improve themselves. In software psychology, I most commonly use code-psychotherapy, better know as "listening to the pain".
Software psychology needs to be done in the real world. We need to listen to the systems that are doing the actual work. Triaging in a test environment is like taking a person to a tropical beach, allowing them to relax, and then asking them what is wrong with their life. There are bizarre corners of the net where odd things happen. If we aren't listening then we're missing the oddities that would never even occur to us while building a test case.
What is a Code Scream (as opposed to Code Smell)?
A code scream is a behavior indication of a deeper problem in the system. It differs from a a code smell in that it is a behavior rather than structural, e.g., an overly long response time rather than an overly long method.
The code can also scream about other things. It may not be a technical insight. It may be trying to tell you something about the business. Since all businesses have turned into software businesses, these software insights are available in every organization.
The data exists - but it is currently not optimized. The real world makes listening hard. The steps:
- you have to learn how to get the data out
- the query runs against production
- the query will take overnight to complete
- it will take a few tries to get it right
The problem with this is similar to how Amazon's software (pre-2000) used to be. At the time it was one executable written in C (everything from warehouse to storefront, you would just start the same exe with different command line arguments). It took 12 hours to compile. As a developer, you would get one update a day, regardless if you were willing to work extra hours.
That is why Amazon moved to a software service architecture. Likewise, listening needs to be more rapid than a nightly data warehousing dump. Collect, store, and present ASAP.
Even better, don't just listen. Have a conversation. How do you do that?
- Ask good questions (to ask good questions (and get good answers) you need to ask a lot of questions
- Collect all the data, because you'll never know when you'll need it. allows for new questions as your desire for insights deepen
- NOW, ask lots of questions - if you get fast answers, you'll ask more questions
- enable the business to "listen to your code"; build dashboards, share, and help others become data psychologists
