If specialization has been the driving force behind economic growth for the last three centuries, why is scrum successful in making software companies more effective, by telling them to move away from division of labor? Given that people like Jeff Sutherland want to apply scrum and agile management techniques outside of just software development, I started noticing more and more disconnects not only between scrum and management theory, but even old-school economic theory.
As a side note, please accept my apologies for using lots of long words and nominalizations (there you go). Theoretical economics jargon is full of them. I realize they’re usually just big names for simple common sense concepts, but they’re precise.
The neoclassical argument for specialization
Here’s a quote from Adam Smith himself in The Wealth of Nations in 1776, effectively describing how a 4800x increase in productivity comes from specialization:
“To take an example, therefore, from a very trifling manufacture; but one in which the division of labour has been very often taken notice of, the trade of the pin-maker; a workman not educated to this business (which the division of labour has rendered a distinct trade), nor acquainted with the use of the machinery employed in it (to the invention of which the same division of labour has probably given occasion), could scarce, perhaps, with his utmost industry, make one pin a day, and certainly could not make twenty. But in the way in which this business is now carried on, not only the whole work is a peculiar trade, but it is divided into a number of branches, of which the greater part are likewise peculiar trades. One man draws out the wire, another straights it, a third cuts it, a fourth points it, a fifth grinds it at the top for receiving the head; to make the head requires two or three distinct operations; to put it on, is a peculiar business, to whiten the pins is another; it is even a trade by itself to put them into the paper; and the important business of making a pin is, in this manner, divided into about eighteen distinct operations, which, in some manufactories, are all performed by distinct hands, though in others the same man will sometimes perform two or three of them. I have seen a small manufactory of this kind where ten men only were employed, and where some of them consequently performed two or three distinct operations. But though they were poor, and therefore but indifferently accommodated with the necessary machinery, they could, when they exerted themselves, make among them about twelve pounds of pins in a day. There are in a pound upwards of four thousand pins of middling size. Those ten persons, therefore, could make among them upwards of forty-eight thousand pins in a day. Each person, therefore, making a tenth part of forty-eight thousand pins, might be considered as making four thousand eight hundred pins in a day. But if they had all wrought separately and independently and without any of them having been educated to this peculiar business, they certainly could not each of them have made twenty, perhaps not one pin in a day….”
This was of course the ideological underpinnings of the industrial revolution, and a move away from craftsmanship as the most effective/efficient form of production, towards factories. Small family run farms (like those in Plymouth, MA) where everyone wasn’t specialized gave way to massive specialized factories that output lots of widgets. Global economic statistics reflect this specialization fueled growth:
From the year 1000 to 1820 the advance in per capita income was a slow crawl — the world average rose about 50 per cent. Most of the growth went to accommodate a fourfold increase in population. Since 1820, world development has been much more dynamic. Per capita income rose more than eightfold, population more than fivefold.
One of the key elements of this growth was the increasing specialization introduced by factories in the theory above. Many people built fortunes putting this insight into practice in factories and agribusiness, as global GDP exploded.
So what’s the problem?
Interestingly enough, there are some similarities between the principles of running small pre-industrial revolution family farm and the principles of agile software development:
Everyone on the team knows and trusts each other since they were family, where individual interaction was more important than processes and tools
Everyone is focussed on a common and clearly defined goal, producing food for consumption or sale
Customer collaboration was more important than contract negotiation, as food was often sold locally due to underdeveloped transport (relative to current standards)
Everyone had to responding to change, as the weather and market conditions was rather indiscriminate and impersonal
Despite following the principles of 21st century agile software development by avoiding specialization, economic growth didn’t occur for centuries. It didn’t occur until specialization became part of the economic system.
The obvious question is why do we think agile software development is economically different from these subsistence farms from the 16th century? Why didn’t that give us business growth on the scale that the above did? The family run farms without specialization were around for hundreds of years, but showing very little economic (business) growth. What’s different now?
There are a number of case studies on the net, many big companies adopting scrum, and a cottage industry of consultants advocating a philosophy that is contrary to one of the key underlying assumptions of economics. More interestingly, much of the data they gather seems to confirm a significant increase in productivity even a 15x growth in function points generated per month, thus showing that the theory of specialization breaks down in this case.
The Agile case against specialized silos
My question for the gods of Agile is: how or at least does de-specialization work in the context of software? I suspect it might have something to do with the nature of software itself, especially how it is produced.
Software is neither a good nor a service, in a traditional sense. In the relatively forgettable book from the dotcom era Blur by Ernst & Young consultants David and Meyer, they come up with one really novel idea. Software has characteristics of both a good and a service. More uniquely, it can ultimately serve as either. This has happened because of the speeding up of the economy due to software.
Speed–the acceleration of business in every respect–is driving products and services to resemble each other in various ways. For one, product life cycles are a fraction of what they used to be. In turn, rapid obsolescence demands continual upgrading or replacement. There’s no such thing anymore as selling a product to a customer and forgetting about him. The people who are your curstomers today will be customers again in six months–if not yours then someone else’s. When you’re dealing with the same customers with that same frequency, doesn’t it begin to qualify as a service business?
Conversely, the need for speed is causing a lot f people in the service business to feel as though they’re selling products. Customers want their services fast, whether they’re bying a burger or financing a home. Under the pressures of time, there’s often no way services can be totally tailored for each customer. At their core, even when services are mass customized, they still have to be standardized, modularized, packaged, and embedded in software. In other words, they have to look like products.
Of course, they were manically trying to sell e-business consulting. To be a little more specific about the difference between goods and services:
Goods are tangible, and transferable while the services are intangible and non transferable.
Goods are separable, and non-perishable while services are inseparable.
Goods are homogeneous while services are heterogeneous.
Basically, software is all the above. It’s also information, codified into a process. It certainly defies easy categorization with traditional classifications, as the authors of SOPA, PIPA, and ACTA have found. This conveniently leads me to believe that working in the software industry, i.e. creating software, may actually be subject to new “rules”, even though Wikipedia thinks otherwise. Yet because you can copy, configure, and modularize software, at any given moment, so it can serve a customer as either a good, a service, or a source of information.
The scrum and agile argument against specialization pushes delivery of working software as the end goal of software creation. This happens because of empirical feedback loops, continuously inspecting and adapting the output to learn as you go when creating software. The end result is a piece of code that works technically, and ideally matches the customers’ needs. Ultimately, the customer typically doesn’t care whether he’s buying a good or a service, he just wants his problem solved. Feedback loops work, because of rapid learning.
Feedback loops break down, when there is insufficient communication among team members, which happens in most software companies due to specialization. First, the Business Analysts specify the functionality. Then the Developers develop it. Then the Testers test it. Then it’s released. It’s the chinese whispers approach to software development, forced on us by the fact that each specialist has to work on the “pin” at each stage of the process. The key difference, compared to Smith’s pin factory, is that everyone creating the software needs to learn exactly what the problem is, and then needs to discover the best possible technical solution to the problem. This information also needs to be disseminated across all departments to the relevant people for the particular feature to work as expected. Just explaining it makes me tired.
Another problem with this approach, according to agilemodelling.com, is that “If you freeze requirements too early, you won’t build what people actually need, you’ll build what they initially thought they wanted.” This specialization implicitly forces us to commit to a particular vision of the feature, before it’s even developed. You lose flexibility before you even write a single line of code. This should be expected, although in an over-structured environment, this kind of requirements churn can exceed half the time allocated to development.
From an economics standpoint, this communication breakdown is the typical effect of bureaucracy. The recently deceased William Niskanen argued in his Bureaucracy: Servant or Master? that bureaucrats do what they can politically to maximize budgets in large organizations, and thus their motives are not necessarily the same as that of the organization. One of the typical strategies includes hoarding information within departments, in order to make their political struggles easier. This is particularly important if that information is needed to produce software effectively. Once specialization creates political boundaries within organizations, its effects clearly break down.
What does this mean for the Agile practitioner?
One of the key inputs in creating software is information. Good information can be gathered individually, but it’s much more effective to share information amongst everyone working on a specific feature continuously. If information flows break down, for example, as a result of over- specialized information-hoarding bureaucrats, then this lack of information will overwhelm any benefits gained from specialization. I would even say that this can create lean-style resource deadlocks, resulting in non-linear losses in productivity.
De-specialization, particularly if it weakens internal political struggle, helps improve productivity. Teams with high communication saturation are much more effective at accomplishing tasks. The firm’s focus is creating value for clients. If at any moment it loses that focus, for example wasting time in politics amongst point haired bosses, then the specialization is counter-productive.
I realize there is much more to scrum than just getting rid of specialization and organizational silos, but this is probably one of the more shocking first changes for companies working from the assumption that specialization is a good thing. What do you think?
Lately, I’ve been exploring a little known corner of high performance computing (HPC) known as FPGAs. Turns out, it’s time to get electrical on yowass (Pulp Fiction reference intentional). You can program these chips in the field, thus speeding up processing speeds dramatically, relative to generic CPUs. It’s possible to customize functionality to very specific needs.
Why this works
The main benefit of FPGAs comes from reorganizing calculations. FPGAs work on a massively parallel basis. You get rid of bottlenecks in typical CPU design. While these bottlenecks are good for general purpose applications, like watching Pulp Fiction, they significantly slow down the amount of calculations that you do per second. In addition to being massively multi-parallel, FPGAs also are faster, according to FPGAdeveloper, because:
you aren’t competing with your operating system or applications like anti-virus for CPU cycle time
you run at a lower level than the OS, so you doing have any protocol stack communication delays
Show me the money
The net result of removing these bottlenecks, you ask? 100-300x faster calculations of portfolio risk.
Now I put that up as an example, just so that I can watch it a few times, so that eventually some of that smartness starts sinking in. The good stuff starts around 21:50, where the mastermind behind the application of FPGAs to credit trading explains how it worked.
Why It’s Worth Caring
It’s not hard to imagine why this much speed would be a competitive advantage. Turning an overnight risk calculation into something you can run in 4 minutes, gives you a better sense of what you are doing, especially compared to competitors. Maxeler, the company who helped JP Morgan get this system going, implies that the speed up was around 30x faster in production.
According to the case studies at WallStreetFPGA.com, FPGAs can also be used for
If you want to hear more check out, the discussion with Wall Street FPGA at the VIShots podcast.
Where I Got More Information
When you are fighting for milliseconds, and all of a sudden a technology speeds up calculations by an order or two of magnitude, it’s hard to completely dismiss it as a fad. The problem is: (wait for it) coding these things is really time-consuming. You’re really getting down to the level of creating circuits in a chip, except you are doing it with code. In this situation, the weapon of choice historically has been VHDL or Verilog, although customized variants of C like SystemC are available to code them, for example at SCLive. There is apparently even an open source hardware movement, centered around opencores.org. Interestingly enough, FPGAs have been around for decades, but only recently have they been seriously considered an option in finance, in addition to other chip level approaches e.g. GPUs.
Personally, I’m interested in playing around with this in a Windows environment for the time being. If you have any specific recommendations on books or chips to buy please include them in the comments, particularly for someone coming at this from a Windows C++ background. So far, I took the plunge and bought FPGAs: Instant Access. It’s a good intro for an electrical engineer, so I am going to dig into Hardware-Software interfaces first. Interestingly enough, all of my low-level C++ started looking a lot easier to understand since I started researching FPGAs.
A hedge fund is a specific type of investment fund, focused on generating absolute returns, often using various types of leverage and investing in publicly traded instruments. Absolute returns mean that the funds focus on generating a specific return, regardless of what stock market is doing. Most investment funds’ results are compered to a relevant stock market index’s performance; however, hedge funds’ goal is to generate as large of a return above zero as possible, regardless of what happens in the markets. While there are many strategies which hedge funds use to generate these absolute returns, they generally have a common goal of generating absolute returns, and that is a key distinguishing characteristic.
Absolute return is typically expressed as a percentage gain of capital on the total amount of capital available for investment. This is typically reported in annualized time frames, so that it is easy for investors to compare against other possible investments. While often associated with hedge funds, you can also earn absolute returns in more “traditional” ways, such as a savings bank account. The difference lies in hedge fund’s goal to maximize the amount returned to investors regardless of market conditions, and also that they obtain these results by investing the money (as opposed to lending it like a bank).
Once hedge funds find an investment opportunity that generates a good absolute return, they use different kinds of financial leverage to increase or decrease the amount of risk they are taking. For example, an individual investor buys an apartment with a 5% down payment. Although he pays for only 5% of the value of the apartment, he is exposed to the risk of owning the whole apartment. If its value goes up, as he hopes, then he earns the full financial “effect” of owning the house using only 5% of the cost of the house. So the investment is “amplified” 20 times. This works both ways, unfortunately. If the house falls in value, the investor loses money at a rate 20 times as fast. A mortgage is a good example of financial leverage, in real estate of course. Hedge funds do similar things, but buying stocks, bonds, and other instruments instead of apartments. Once they have identified a good opportunity, typically of a lower risk, they use financial leverage, attempting to increase the returns to their benefit.
As a result, their overall return profile is quite different from other types of investments, making them a very attractive part of a balanced portfolio. Because they are leveraging investments that are typically not related to the standard indices, they are very good at “rounding out” a portfolio-in theory. In practice, there were significant concerns that many hedge funds performed badly during the instability of 2008-2009, when the stock market was crashing, so not all of them were successful in generating the absolute returns for their investors.
How shorting generates absolute returns
In the hedge fund sphere, absolute returns are typically the result of a combination of both long and short strategies. A long strategy is a traditional buying cheap and sell dear approach. A long strategy is the typical approach anyone uses in investing, but also just when shopping. First you look for a good bargain, and then hope to sell it after the price goes up over time.
A short strategy is the exact opposite, where you sell dear and buy cheap. If dealing in a physical asset, this is like borrowing a specific valuable thing, selling it (despite not really owning it), expecting the price to fall, so that you can buy back the same thing in the future for a lower price. Once you have the same thing back again, you can return the valuable to its original owner. Here you make money based on how much the price of the valuable FALLS. At its core, a short strategy is any strategy where you make money when the price of something falls. There are multiple ways of entering into short positions, but most often this involves short selling of stock, one of the two currencies of a currency pair, or derivative instruments such as options or futures.
When entering into these various positions, hedge funds typically look at correlations among groups of instruments. By owning positions in two instruments, which are negatively correlated to each other, a hedge fund is able to generate absolute return. If a hedge fund owns two different but related instruments, with one expected to rise in value, and the other to fall in value, a hedge fund will often take a bet on this convergence. If the trade works out, the hedge fund makes money on both sides of the investment, once the prices converge.
A good example is the pair trades strategy. A trader believes two stocks in one industry are significantly out of line with the valuation for a particular industry, and he expects of them to converge to the industry average. Let’s take Apple (AAPL) and Microsoft (MSFT) as examples. The trader buys long the instrument which he believes is relatively inexpensive, for argument’s sake is Apple, and simultaneously shorts the instrument which he believes is too expensive, say Microsoft-simulatenously. The trader he creates a net position, which has completely different characteristics than just investing in either stock. The net performance is independent NASDAQ’s performance, probably the most relevant index to these two stocks.
The two instruments are effectively combined into one position, a pair. It is similar to a spread. The trader expects the spread to narrow, and by taking equivalently sized positions of both instruments, he can earn an absolute return, if the two stocks converge as he expects. He can amplify the convergence effect with leverage. Let’s say the trader estimates the two stocks (Apple and Microsoft) will converge one percent. Using leverage, e.g. options, stock loans, or other instruments, the trader can multiply his exposure using the same amount of money. So using a fixed amount of money, he can create a position which has a risk profile which is the equivalent of e.g. five times the amount he could buy outright. In effect, this is the same as the apartment investor above, but instead the investment is in a spread between two stocks.
The above is only one example of an absolute return strategy, a simplest case to illustrate a point. Typically, a hedge fund will employ a large number of strategies within their particular area of specialization or asset class. The funds under management are allocated to the most lucrative expected opportunities, while managing the overall risk of the portfolio. For example, by actively looking at the proportions between net long and net short positions, hedge funds can choose how much they aim for pure absolute returns.
Absolute return strategies themselves typically have a low correlation with markets. For example, even though a specific hedge fund invests only in US stocks, it’s returns will typically not follow a stock market index like the S&P 500. Buying a hedge fund and buying an index fund would give very different results, assuming you are buying both using a long-term buy and hold strategy. Correlation is a statistical measure of the strength of the relationship between two variables, and combining investments that have a low correlation tends to increase returns and/or lower overall portfolio risk.
Leverage, the two-edged sword
Another major innovation relative to other types of investments, is the use of leverage when investing. Because hedge funds often use various types of loans or financing, and they frequently go after investment opportunities which generate absolute returns, they can multiply their earnings many times.
On one hand, a hedge fund aims to follow a series of strategies, which exceed the interest rate at which they need to pay financing costs. This allows them to invest in more “boring” instruments or strategies - ones which give a very steady return-but amplify these strategies’ effects. This only works when the strategies’ returns exceed the funds’ borrowing costs. On the other hand, if they are on the wrong side of the bet, this leverage amplifies their losses. Not only did they lose their own money, but they also lose borrowed money, which still must then be returned.
Leverage is strongly tied to the concept of position sizing when trading. An important part of the trading process is determining how much money you are willing to risk on a particular idea. If you don’t put enough, you won’t get very much out of your analysis, regardless of how good it is. If you put too much, you risk putting yourself out of business completely, or blowing up. Leverage helps a trader get the most out of an idea, while allowing him to fine tune the proportion of his portfolio’s exposure to a particular risk driver.
Largest funds, 2011
The table below lists the largest hedge fund, as ranked by Assets Under Management (AUM):
Based on recent estimates of the Hennesee Group LLP, investors in hedge funds are classified into the following categories, in about the following proportions:
Historically, most of the capital provided to hedge funds came from wealthy individuals, but over time, more and more funds have come from institutional investors. Average consumers often unwittingly invest in hedge funds, via their pension fund’s activities.
Strategies
The table to the left, from Hedgefundresearch.net and Pertrac, lists the highest performing strategies existing the hedge fund space. It is a good starting point for an analysis of strategies which hedge funds follow, and which worked. You can compare this against S&P 500 results during the same time period to get a sense of how effective these stragies are; however, these are the best performers, so they are not necessarily indicative of typical results obtained by hedge funds.
See the Hedge Fund Research classification for more detailed information, especially a definition of the strategies which hedge funds use. Also you can check out the latest Hedge Fund index results on HFR’s site. Also, there is a more specific breakdown of historical data up to 2008 at Hennessee Group. This information was once freely published on the web; however, today detailed index performance information is available only to paid subscribers.
Note that in 1997 and 1998, hedge funds as a whole underperformed the stock market. Because HFs focus on absolute return, investors get better returns by money directly into the stock market during a strong bull market. Conversely, notice that the returns generated in the highest return strategy type was consistently positive. For example, in Q1 2009, most equity markets have suffered negative performance across the globe; global macro hedge fund strategies, as well as a number of other strategies, have earned a positive return in these difficult times. Looking at the above, you will also see that slowly the returns of the sector as a whole are declining, as competition is increasing among hedge funds because the amount of money they manage and invest is increasing rapidly, thus making the sector as a whole less attractive.
Useful links
www.seekingalpha.com: an aggregator of blog posts about investing, with some interest in absolute return strategies
After having worked in an Agile environment for two years, I realized that though Agile’s productivity gains are rather paradoxical, even counterintuitive, particularly for traditional project managers. Paradoxes are counterintuitive gems, especially for software teams. They hold significant wisdom, even though they are often easy to express.
In particular, Agile project techniques take advantage of a number of paradoxes. After taking a closer look at the more popular agile methodologies, I think each has its merits. The best possible approach is to use all of them simultaneously: XPishLeanScrumKanban. Ha! Well, kind of.
Each style has its stronger points, and because they are similar enough, you will probably get the most value from combining them. When combining them, you risk missing what each one tries to achieve, or how it relates to agile fundamentals. The end result is a mess. While I work in a scrum environment, it is interesting to think of potential performance boosts from incorporating elements of the other approaches. Here’s what I think is the most worthwhile in each approach:
Scrum
Scrum focusses on delivering working software every 30 day iteration (or shorter). The short interval gives product teams greater flexibility, to adjust team priorities, while still getting high quality product at the end of a sprint (iteration).
From a business point of view, the above aspect of Scrum turns a programming project into a “real option“. The project generates more value because of the newfound flexibility. Traditional waterfall projects that plan everything up front straightjacket teams before they really understand the problems they are trying to solve. It’s better to be more flexible, and incorporate changes based on what you learn from inspecting a problem domain more closely. This prevents spend months of time on work which is either not needed or not useful to the end-user. While the other agile processes also promote an iterative incremental approach, scrum most clearly promotes this approach in my mind. Basically, this flexibility has a very real value for stakeholders, particularly since product life cycles are shrinking in most industries, including finance.
The effectiveness of scrum depends largely on how clear the team’s “definition of done” is, and how well the team holds itself accountable to it, according to scrum’s co-creator Ken Schwaber. The “definition of done” includes all criteria for accepting that a team “did” a requirement (user story). The requirement is then officially ”done”. Typically, this includes not only development, testing, but also any work required by the end-user, such as documentation.
When really following pure scrum, according to the spirit of the scrum guide, scrum co-creator Jeff Sutherland claims productivity goes up about 5x, measured by the amount of extra revenue generated. If combined with other approaches, like XP, it can go up 15-20x. This depends on a number of factors potentially external to the team, such as sales. The numbers are roughly analogous to the software “function points”, generated by the teams tracked in his studies. Function points are a traditional unit of measure of functionality, reduced to a least common denominator, which were used to track productivity of software developers in the 1980s, thus allowing for a reasonably accurate comparison of scrum output against traditional waterfall approaches.
What I found initially counterintuitive, was that planning the long-term future was much less valuable, than planning the immediate future in detail. If you spend relatively more time on short-term planning, your long-term horizon will take care of itself. Of course it is good to have a long-term project vision or goal, but beyond that it makes little sense to go into minute details what the fifth month of a project will look like. Instead you can get the biggest gain by knowing in extreme detail what you will complete, and how you will complete the most important tasks in the first month. You ensure you get great results now, particularly since scrum focusses on delivering working software which can immediately generate business value. Moreover, the flexibility you get is liberating. You can even estimate a financial value of it if needed.
Extreme Programming (XP)
In my opinion, XP’s greatest insight is that certain technical practices eliminate the need for technical documentation; instead you get well designed software in an executable specification, which itself is a deliverable, with a very low amount of bugs, assuming you write decent tests. Teams plan and design on a “just-in-time” basis, instead of preparing a big “up-front” design document. This allows you to learn about the problem domain and the technology you are using as you go. XP also stresses managing explicitly by goals instead of tasks. You are extremely focussed on being effective, i.e. you don’t spend a single minute on working on functionality that isn’t required now. The four key technical practices are refactoring, TDD, pair programming, and continuous integration.
Back in the late 90s, I remember the 800 page requirements document that a customer gave my team, on a railway transit project. We then created a specifications document that was 300 pages long, and a page-turning series of design documents about how we were going to create a piece of software before even writing a line of code. This was pretty much standard practice at the time. Thinking about it now in the context of what refactoring, TDD (or at least unit and integration testing), and CI do together, it was all mad.
The most important of these practices is refactoring in my opinion. Refactoring means constantly improving the structure of the code as you work on it, so that it stays as flexible and easy to work with, at the moment you started. The more clearly you express concepts in code, at least compared to how people think, the easier you can continue working productively with it. A large code base that isn’t refactored tends to accumulate “technical debt”, which is essentially “borrowing forward” time. If this gets out of hand, the software becomes very hard to work with. The productivity levels of developers can become very low. Moreover, most developers will avoid making new changes to unrefactored code, due to fear of unexpected side effects. In contrast, if code is well-refactored, it is easy to introduce changes into code, as it is pretty clear what the code is meant to do. It’s the software equivalent of cleaning up your kitchen after you’re done using it.
I haven’t seen explicit productivity numbers for XP on its own. Scrum combined with XP technical practices can give up to 15x productivity gains, even when globally distributed. While this is not isolated, it does give some sense of the business value of this approach. XP itself, though, is hard to do. It requires certain technical skills such as TDD, which is difficult to learn because the knowledge is very context sensitive. For example, it’s much easier to do this on Web technologies than large legacy C++ projects. It also requires a willingness to stick to the process, which is quite difficult. The benefit is that you get a massive amount of feedback as you go, which enables you to constantly course-correct continuously.
XP taught me that spending time on refactoring saves time overall. Once a concept is clearly expressed in code, it becomes easier for anyone to maintain it in the future-whether it’s you or someone else. Jangled code wastes everyone’s time, and expresses a lack of clarity in thought about the problem. Moreover, you can supposedly see indicators of most of a team’s dysfunctions (or lack thereof) in the code they maintain, based on how effective they are at refactoring. Well designed software should be easy to understand by any new developer added to your team. Good software design follows how we think-as people-about the problem we are solving. In contrast, the compiler doesn’t really care how you structure your code, as long as it can compile it.
Kanban
Moving slightly outside of the realm of pure software development, Kanban helps teams self-manage by increasing visibility of problems. Kanban has its roots in a manufacturing environment, where work is more sequential than typically in software development, but there are more parallels in a software maintenance environment to make it using worthwhile. Kanban helps make problems more concrete and visible using “cards” which represent issues, so that the team can move them through various “swim lanes” or stages of a process. There are no prescribed explicit structures, like scrum’s various meetings, as the team decides what it feels is best and acts accordingly.
Personally, I think the most valuable aspect of Kanban is the focus on visibility. Traditionally project managed projects hide a lot of complexity behind a big stack of documents, Gantt charts and risk reports. Paradoxically, the more documentation that is produced, the harder it is for any member of the team to actually know what is going on–in totality.
Kanban forces the most important elements of a specific project to be constantly visible for everyone, not only the team, but also management. It helps focus on the biggest impediments, as removing them tends to provide the largest jumps in output. It feels uncomfortable to face the impediments, but they are typically bottlenecks on performance. Simultaneously, Kanban is flexible enough to accommodate for any kind of problems. As a result it helps everyone focus on the most important issues at any given time, because of the visibility it generates. The board where the cards are placed visually constraint the amount of things available. Even more importantly, the work in process is always limited to a small amount – typically 4. This accommodates for the small “working memory” we have as humans, where we are only capable of effectively processing a small number of concepts (or problems) at once. This approach also minimizes the effort required to switch between tasks. By keeping the right amount of high priority items in front everyone’s face, Kanban prevents wasting time by getting lost in unimportant details. It’s immediately clear to everyone involved what the priority is, which makes it easier to actually “do” it.
My personal experience with kanban is in my Kitchen. After having read Jim Benson & Tonianne DeMaria Barry’s Personal Kanban and David Starr’s Agile Practices for Families, I tried implementing Kanban to keep track of what goes on around the house, and what we need to do during a week-long iteration. It…kinda worked. Although it’s good to put everything up on the board, often I don’t feel like going ahead with the next work item. After all, I’m not at work. I want to relax at home. Although I do have to admit, my wife and I got through a major project that had hung over us for years when we initially implemented Kanban. You could say that we have had some success with it.
Kanban speaks to my belief that transparency helps clean up a lot of problems. As long as you can’t see what is going on, anything can be happening. It’s very difficult to move forwards if you don’t know exactly where you are. In theory, Kanban can be especially useful in a highly politicized situation, where people are more concerned with looking good, than with making progress. Kanban makes it clear what the most important problems are, and what discussions really need to happen. Once this starts, everyone can focus on the current key issues. Over time, you reduce internal politics, and get a massive spike in effectiveness.
Lean
“Remove waste”. That’s the very simple message Lean promotes, even though the methods of waste removal can be quite complex. By getting rid of waste, you are capable of generating much more using the same inputs. Lean tries to separate the activities from the goals they are addressing, and figuring out if the same goals can be achieved in a better way. Traditionally, lean came from manufacturing, but the basic ideas can be transplanted into an office environment as Ole Dam says, even a software environment. Tom and Mary Poppendieck have trumpeted about the benefits of lean in software for a long time.
The most useful counter-intuitive insight I’ve discovered in Lean was the effect of removing a bottleneck. Eli Goldratt in his fantastic educational novel with a cheesy title, The Goal, points out that removing one bottleneck in a system has dramatically non-linear effect on the output of the whole system. Goldratt illustrates basic Lean principles, while telling the story of Alex Rogo, a frantic manufacturing plant manager who needs to rapidly turn around an ineffective plant. There are a number of parables woven throughout the text, such as how the slowest person on a hike (subsystem) determines the speed of the whole expedition (system).
Many of these insights have a direct implication for personal life also. At any given moment, removing your biggest bottleneck should be your only priority. A top priority, by definition, is only one. Every other activity you can undertake will have significantly less impact on your life than addressing this particular problem. Therefore, you will be most effective if you concentrate all of your resources on this top priority: time, money, effort, the soul of your firstborn child. While it may initially feel more risky, in fact you will get much more accomplished with fewer resources.
If you want to have a listen to someone discussing lean software on your next commute or “fun run”, try this eposide of Herding Code: BDD and Lean Software Development. Some of the comments by Scott Bellware are intentionally harsh, but I think he does manage to make a number of useful points, particularly about Lean in a software engineering context.
Key Takeaways
Regardless of whether you apply this to a software project, or consider the below as pseudo-scientific self help bulls**t, the below are probably the most useful parts of Agile:
Iterative incremental work tends to be more effective than massively detailed plans.
Clean up after yourself continously.
High visibility helps clear problems quickly.
There is always only one top priority: your biggest bottleneck.
Interestingly enough, they are rather “uncorrelated” in terms of how valuable they are. You can simultaneously reap the benefits of all of them.
What about you? Have you tried any of these approaches? What do you think is the most worthwhile?
Health Warning: Massive geekiness alert. The fastest way around Excel is keyboard shortcuts. Shortcuts are like mnemonics, like a quick way to tell the computer what you want to accomplish. You can easily chain them together, like on a Bloomberg terminal. When entered in the correct order, they become shorthand for what you want to accomplish. Knowing how to navigate your way around using good keyboard habits signficantly increases your productivity on this platform.
A hidden gem…whatever is in your clipboard is pasted into the selected cells, but only the values, so you drop any formulas driving the values. Especially useful when using Excel to format data in a particular way, but you don’t want it to change “types” without you knowing, or alternatively when you want to force a change, such as forcing a date to become a string.
In edit mode, switches between absolute and relative references, otherwise like Redo
Adds or takes away the $ before the row and column values, giving you control over what changes and what stays the same when you fill down or right. If you aren’t editing a cell, it works similar to Redo.
The following are the available ways to develop new software on Excel, that are still worthwhile and viable on recent versions of Excel, as mentioned on MSDN and other places:
C API and XLLs: essentially fully integrated DLLs that are accessed as Excel worksheet functions.
VBA: a popular option to quickly generate something which accomplishes a particular busines goal. Good for a small, low resource project, ultimately hard to get fully stable because of inherent language features.
COM: Legacy Windows infrastructure in C++; reasonably powerful, well supported historically by 3rd parties. VBA uses COM to interact with Excel, so it’s going one level lower for direct access.
The Microsoft .NET Framework: Microsoft’s multilanguage platform can also “speak” with Excel via Visual Studio Tools for Office (VSTO). Excel 2002 and later versions can access code resources contained within .NET assemblies.
Direct accces via C#: bypass the weight, and just use excel files as objects using tips from OpenOffice. See the ExcelReader on CodeProject for more information.
I’ve always been curious what the differences in performance are for using the above approaches. Using the following cell function or its programmed equivalents, the blogger Codematic decided to benchmark the various options of how to approach programming things in Excel:
{=INDEX(B1:B104,MAX(ROW(B4:B104)*(B4:B104<>0)))}
The table below summarizes the results.
Performance (Excel 2003)
57,000 formulas, calculated 10 times
Version
Time
Performance
Seconds
% of Excel
Excel
27
100%
VBA
29
107%
VB6
32
119%
C#
260
963%
C++ (XLL+)
13
48%
While clearly the last option is the fastest one, it is worth pointing out that they were obtained with a commercial product that prouduces a custom platform for developing fast XLLs (Planatech), with the original projects potentially available for review. Also, you can just reach for the XLL development tools available from Microsoft, for example the Excel 2007 XLL development kit. So for anything that is performance critical, such as pricing applications, or requiring fast access to legacy systems, it seems the XLL approach is the best available one.
Personally I’m surprised that the C#/VSTO option takes such a hit. Codematic claims it’s related to “passing through the Primary Interop assemblies for Office 2003″, and that the costs may not outweigh the benefits of using this technology. The main benefit of using .Net, is that you can access code written in any other language which has also been optimized for .Net, and many options do exist. My guess is that Microsoft will eventually address this issue,since both Excel and .Net are their projects and directly within what they are able to influence.
Performance over .Net also significantly depends on exactly how you are using the inherent functionality. For example, DotNetPerls compares a few functionally equivalent syntaxes available to programmers. He says “using the Cells[] indexer on a range, or getting the Range with the Range property or get_Range() is very slow.”
From what I understand this has to do with cell value retreival across COM boundaries, according to CodingHillbilly, a self-professed Interop unExpert. Basically, this may be the same issue encountered in VBA , where it is better to interact with ranges and not cells individually if you can avoid it.
Takeaway:
So you have a few options when attempting to code something new against Excel. VBA is a great prototying tool when you need to quickly create something that works, and the worksheet functions aren’t good enough. C# and VSTO give you much greater flexibility to step outside of Excel and integrate it with many other systems and languages. But most importantly, reach for XLLs when you want speed or access to high end server resources.
How do I get Excel to run a macro without showing all intermediate steps in a calculation? So, after I kick off the macro, there are no changes on the screen until the macro has finished…
Answer
This is done via the ScreenUpdating property of the Application object:
Application.ScreenUpdating = False
‘your code
Application.ScreenUpdating = True
This will also speed up the macro, as it won’t waste processing power on displaying the changes. Also, if it take a long time, you may want to add some information in the status bar for users of your macro to know it’s being processed.
How important is the knowledge of VBA and C++ for working as an analyst or as a risk manager in hedge funds.
I would like to develop these skills and bought some books which deal with modelling derivatives pricing in VBA/C++/Matlab.
Do you have in mind a specific book/area I should focus on in Excel/VBA/C++
Thanks for your help.
Answer
Hi gabx,
Good question. VBA and C++ are both useful skills in such roles within a hedge fund, with each typically used in a slightly different way. VBA is probably more for risk management, as hedge funds tend to be small, and a lot of analytical work is done in Excel. In this case, VBA is used to automate various processes such as position and trade reporting, or various types of simluations for risk management purposes like Monte Carlo.
In contrast, C++ is usually used more frequently when dealing with automated trading in various guises. In this case, the code needs to be very efficient and reliable, and C++ is traditionally a very robust language which helps build production quality tools. There are also many libraries that are useful to traders or risk managers which are then easily integrated. Once a good strategy has been identified, C++ developers come in and produce a high quality solution that executes a particular strategy relentlessly. C++ is sometimes used in a risk management function also, for example when looking at correlations within large portfolios. For this to be reasonably quick, it needs to be done in a way that is responsive, and quickly enough that the results are tradeable.
The popularity of both technologies largely depend on the particular type of hedge fund, more than the function being performed within it. For example, statistical arbitrage houses typically have big server farms, and lots of code which optimize responsiveness. In contrast, a risk arbitrage house or even certain long/short hedge funds invest much more based on equity fundamentals, and there the technology requirements tend to be much lighter at many of these funds.
I had a suspicion, that there must be a way to mathematically determine optimal position sizes. Ralph Vince, the author of the Handbook of Portfolio Mathematics: Formulas for Optimal Allocation and Leverage , beat me to the punch. In this volume, he summarizes his previous work, and describes in detail, a mathematical model, which helps calculate optimal position sizes, based on their mathematical values. He goes to one of the core elements of trading. Position sizing significantly affects the amplitude at which returns are generated for a particular position. A good trader needs to have a sense of how much he is willing to risk on a position, and how much he may gain as a result. Transaction costs affect this particular trade-off also, and need to be taken into account. As you cannot be sure that any particular trade will be profitable, how you size your positions can affect your level of success and trading consistency, and also in the extremity of your failure, as for example Long-Term Capital Management (LTCM) learned in the late 1990s. He goes at the topic, though, from the perspective of a trading IT consultant, not as a trader. Even though in theory, he may be right, in practice, I think there are a number of risks which need to be taken account of when position sizing, such as liquidity risk. Most of these factors are hard to model, but necessary to take into consideration when putting on positions.
Vince suggests that every position has an optimal f, where the investor maximizes return over a long time frame. He essentially generalizes the Kelly criterion to the typical situation that a trader faces. The Kelly criterion, developed by engineers at Bell Labs in the late 1940s, in order to resolve the problem of data transmission over long-distance lines, can be reapplied in the context of geometric growth, and how it applies to money-management, as both problems “are the product of an environment of favorable uncertainty”. With some basic back testing, a trader gets a sense of what may happen to a position and whether it is worth putting one on. The values generated during back testing can then be used as inputs into Vince’s formulas. Vince uses an empirical approach based on statistical distributions to calculate the optimal f for a particular position. If trader uses of value below optimal f, he will be generating less returns than he could be, assuming a particular investment is modeled as a time series, generating P&L of a certain mean and standard deviation, i.e. a statistical distribution of returns. If the trader is above his optimal f, he runs the risk of complete wipeout, due to his position sizing. This means that a few losses in a row can completely take him out of the market. This modeling of a particular position as a random walk provides a framework for calculating its optimal size.
With this approach, Vince is essentially talking about the amount of leverage being put on a position. While leverage increases the total amount of return, it can also increase total losses. Assuming a particular position continues to follow a random walk according to certain patterns, patterns which can then be described mathematically, it becomes possible to optimize returns via the size of the position. Because the approach is largely empirical, it is rather simple to also include the effect of transaction and financing costs, and their implications for the position size.
Key takeaways:
traders should strive to maximize geometric, but not arithmetic growth, in positions
the actual approach to position sizing depends on the total value of your portfolio
classical folio construction, Markowitz, can be superseded by newer models, which take into account the properties of the new asset classes, including derivatives
While in theory, he is right, in practice, I think he is avoiding a few important issues. Position sizing based on historical performance implies that all of future performance will follow the same distribution. This is not necessarily the case, and can be hazardous to your health. Also, you cannot assume that you’re operating environment will always be the same, particularly when trading in a bull market. For example, just because it is easy to enter into and exit transactions in a bull market, when sentiments go south, you can be stuck with positions in which you know will continue to lose money for you, but that you cannot sell; therefore, more conservative position sizing is in order.
A quant trader friend of mine at a hedge fund dubbed Hedge Funds: Quantitative Insights by L’Habitant -once a major book in the CAIA program– as “Hedge Funds 101″. This is true of the CAIA program as a whole. The book itself is a good introduction to how the industry works, and how investors evaluate hedge funds.
The rest of the CAIA program expands on these basics. If you are looking at various certification programs, the CAIA program is particularly useful if you are interested in being a better alternative investment professional, or just becoming one for that matter. Last March, I decided to look into the CAIA certification program, as a small challenge to keep myself sharp.
The exam itself covers alternative investments. A large chunk of the CAIA curriculum includes quantitative analysis and hedge fund-specific material. The CAIA exam’s focus on how hedge funds’ clients see them attracted me to the program. I expected the CAIA would help me understand my clients better, because I will understand their reporting needs in greater depth. While I do already have a degree in finance, I expected the CAIA would give me much more depth in hedge-fund specific finance.
What do hedge funds think of the CAIA?
I think the material is probably most useful to people working for or with hedge fund investors. These would include Fund of Hedge Funds, Family Office funds, endowments or anyone else actively investing in hedge funds. The material is rather qualitative, so it’s basically a really good way to get your head around how hedge funds are different from other investments types. There are some quirks, both from an asset class point of view, and how to diversify portfolios containing alternative asset classes.
On the hedge fund side, the material would be most useful for the people actively engaged with investors: Investor Relations and/or Operations. This knowledge would be pretty valuable when raising capital, as you get to understand the typical portfolio-level tradeoffs your clients have, so it’s probably easier to craft a persuasive argument. Later on, when dealing with reporting and relationship management, this background could also be useful.
Is the CAIA useful?
For investors in hedge funds and hedge fund ops people, definitely. Really useful-like preparing small note cards for giving a best man’s speech.
From a quantitative risk management point of view? Probably not, as the exam’s not particularly quantitative. It certainly makes you aware of liquidity risks, though, through some of the case studies and other material. This is probably the most important risk facing hedge funds.
Trading? Even less so. Most of the trading-related material is at the level of being able to understand terminology traders would use. There are probably better certifications and books to read first, if you want to be a better trader. That said, getting the certification can’t hurt your results, and it will make you more aware of the institutional context, which can be useful.
Is the CAIA hard?
Well, it’s not conceptually hard, just time consuming. There is a lot to read and memorize. There are some formulas and quantitative questions. As long as you:
put in 200 hours to master the material at each level (I should know, because I didn’t once and it wasn’t pleasant)
you know beforehand how to add, subtract, multiple and divide, and what a square root is
you’re willing to learn how to use either one of the funny financial calculators
you’ll be ok. Not to mention, I’ve heard it’s easier than the CFA.
Expected personal benefits of the CAIA program
Here are the reasons why I originally decided to take the Level I exam, in anticipation of getting the CAIA certification:
1. Learning is the key to moving forward career-wise. Any permanent change in my life will require that I change myself first. The classic example: lottery winners who lose all of their winnings within a short period of time, because they don’t know how to handle their newfound wealth. Just as they didn’t expand their ability to handle the extra cash, I want to expand my abilities in order to permanently move forward. As a result, I need in the skills imparted in the CAIA exam to ground my knowledge.
2. Keeping the skills fresh is crucial. What’s the point of getting a great degree in finance, and then not using it for a long time? As most of my time at work is spent on the IT side, I wanted to have a good refresher of financial concepts. The CAIA program provides a comprehensive framework for alternative investments, so it was much more than just a refresher. It builds on all of my previous knowledge and expanded it.
3. Education is a great way to respond to a recession, such as the one caused by the credit crunch. On one hand, you get a better understanding of what’s going on around you. On the other hand, you are more employable in a difficult job market with a relevant certification, like the CAIA. When the economy does pick back up, everyone with relevant certifications will be immediately considered a subject matter expert. I remember how the job market looked when the economy was overheating. Every recruiter and their grandmother were willing to hire. The primary requirement was knowing how to google hedge funds. I’m certain that people who were CAIA certified had an additional negotiating edge. You can think of getting any certification in market timing terms. It’s best to get certified before you really need it, while the qualification’s value will be greatest financially during boom times.
4. While my primary interest lies in hedge funds, and the CAIA program exposes me to a number of industries. Private equity, venture capital, real estate, and commodities are also covered. Arguably commodities are closely related to hedge funds anyway. Quantitative methods in the CAIA program are also strongly applied in the context of hedge funds. This variety is intellectually challenging and useful, as it helps think of hedge funds as one of a number of alternative investments. Hedge funds indirectly compete for capital with all of these other alternative instruments, as a result of legal frameworks and industry standards, differentiating them out from mainstream investments. If an institutional investor can invest in one of them, he can invest in any of them. As a result you will be looking at all of them, to seek out the best possible investment given his goals.
5. Given that the hedge fund industry requires keeping up in a number of areas of knowledge, such as finance, IT, mathematics, in addition to all the usual business skills, recent certifications’ content are usually seen as signals of interest in the industry. As I have a strong finance and IT background, but mostly work in IT, the CAIA certification will help me move back towards a greater business focus. In general, certifications help to change job roles.
6. In theory, the more you know, the more you earn. While in my case my employer is funding my participation in the CAIA program, and as a result will probably be unwilling to give me a raise upon completion, my indirect value on the job market will rise. Should I ever wants me to change jobs once I do have the certification, I can expect to earn that extra bit more. Unlike the CFA, I don’t recall ever seeing any hard numbers on this particular topic yet, with respect to the CAIA program.
7. I like how the CAIA program has been standardized enough, that the CAIA association can estimate the total number of hours (150,200) that it will take to prepare successfully for each level of the exam. It don’t think the material is conceptually difficult. It just requires a sufficient amount of time to master their required knowledge. This makes it easier to schedule, and to carry out.
8. Especially since I am quite comfortable where I am, in the CAIA program helps me grow, by challenging myself. I expect the main benefit of the CAIA program will be certain critical thinking skills, the greater precision in how I approach problems, and how I solve them. Much of the material refers to things I’ve seen before, but now my knowledge will be systematic, giving me a better sense of relative importance of each bit of information.
Ok Luke, What’s The Bottom Line? Is It Worth Doing the CAIA?
It costs a lot of money if you are paying out of pocket, but if you are serious about hedge funds, learning the material in this exam will help you understand the most important part of this industry: their clients’ perspective.
My company has generously been sponsoring my examination process, so I’m sure that helped me decide to get going. If that wasn’t the case, I’m not sure I would go ahead with it. I guess everyone needs to evaluate the cost-benefit ratio for themselves. Although the cost is admittedly high, it depends on exactly what your current or expected role in the industry is.
Then again, as Derek Bok a former president of Harvard University said, “If you think education is expensive, try ignorance.”
Oh, and if you’re into hedge funds and automated trading, you can follow this blog by entering your email above.
Join Lukasz Szyrmer to discover more about hedge funds, algorithmic trading, and agile software development. This blog's author, Lukasz, works in the hedge fund industry as a software developer. He has a MSc in Finance, and no formal technical qualification of which he's aware. Read more
The table to the left, from Hedgefundresearch.net and Pertrac, lists the highest performing strategies existing the hedge fund space. It is a good starting point for an analysis of strategies which hedge funds follow, and which worked. You can compare this against S&P 500 results during the same time period to get a sense of how effective these stragies are; however, these are the best performers, so they are not necessarily indicative of typical results obtained by hedge funds.
by L’Habitant -once a major book in the 
