ETF Trade Execution Quality during the Flash Crash Month

Subtitle: Some market orders received better price execution than limit orders.

Recently, all market centers disclosed Rule 605 reports for May 2010. This data provides insights into the quality of trade execution across all market centers by ticker, order size, and order type. The following analysis focuses on a favorite focus point of the Flash Crash debate: ETFs. By utilizing the Rule 605 disclosures for May 2010, a volatile month for equities trading, we can observe where ETF trades received better price execution.

There are no clear strong or weak points apparent in the numerous graphs below. Each reader may have an interest in specific ETFs or market centers. Unfortunately, there are too many permutations of ETFs and market centers to be covered within this post. The following graphs sample a few ETF complexes and market centers with the highest trading volumes during May 2010.

More importantly, Rule 605 disclosures summarize trades for the entire month, not itemized by day. Hence, the following graphs do not solely represent trading activity on May 6, 2010, when many ETF orders were cancelled as a result of extreme price volatility resulting from brief market illiquidity. (Judging from the wide range of price improvement statistics, these cancelled trades appear to be included in the data, but one cannot assume the all market centers included erroneous trades in their disclosures.) For some ETFs and specific market centers, the price volatility on May 6 may have been large enough to significantly impact the summary statistics for the entire month. Nevertheless, a comparison of the same statistics across ETFs and market centers should demonstrate how trade execution performed on a relative basis.

In previous posts (iShares/NFS and Schwab/UBS) concerning ETF trade execution quality, Fundometry studied how much trades were executed inside the NBBO, at the market, and outside the NBBO for specific market centers. One post defined a useful summary metric called Weighted Average Price Impact ("WAPI"). In the Rule 605 disclosures, price impact (in dollars per share) is disclosed separately for trades where price improved and degraded (or, as some might say, disimproved) relative to the subsequent NBBO. The overall price impact of trades can be summarized by averaging the price impact for individual categories of trades, based on improvement or degradation relative to the NBBO and weighted according to respective volumes in each category. A positive WAPI indicates that overall more shares traded with price improvement (i.e. price at execution was better than the NBBO when the order was submitted) than with price degradation, and the opposite would be implied by a negative WAPI.

In addition to WAPI, the following graphs plot volume of executed orders, including orders routed to other market centers. Ideally, this data on trade execution would be itemized separately for internal orders and orders routed to other market centers (in cases where the best price existed at another venue). Unfortunately, the data summarizes trade execution quality across all orders executed regardless of venue (internal or external). In order to highlight the impact of orders executed internally to any given market center, only data is included if the volume of internal orders contituted at least 80% of total executed volume. (The final graphs in this post follow a stricter threshold of 5%.)

Finally, both WAPI and volume are broken down by market orders and marketable limit orders - the only two types of orders for which price improvement/degradation statistics are available.

Based on the Rule 605 disclosures sampled in this analysis, three ETF complexes had the largest trading volumes of market and marketable limit orders during May: SPDR, iShares, and ProShares. Initially, ETFs will be summarized at the complex-level (volume-weighted summary of all tickers within an ETF family) across different market centers (exchanges, ECNs, ATS, etc). Subsequently, selected individual ETFs will be analyzed in greater detail, by order size and order type. (Given the number of graphs involved, including other ETF complexes or selecting more individual ETFs would cause this post to become rather voluminous.)

Complex-Level Perspective

In these complex-level graphs, market centers displayed along the horizontal axis are sorted according to overall WAPI. Market centers which executed orders with overall price degradation (outside the NBBO recorded at the time the order was submitted) are displayed closer to the left side. Despite this sorting order, some market centers achieved notably better or worse trade execution quality depending on the order type (market vs marketable limit).

The following graph summarizes trade execution quality for SPDR ETFs during May 2010.

For the SPDR family of ETFs, a few low-volume market centers exhibited the least favorable trade execution quality. Among market centers with notable volume, National Financial Services (NFS), BATS, and Knight Capital Markets performed less favorably relative to their peers, in terms of WAPI. Depending on the market center, market orders received better price execution than marketable limit orders (as seen when red diamonds are higher than blue diamonds). While there is no clear indication of where market orders should have fared better, marketable limit orders (a buy order above the best offer or a sell order below the best bid) did not always result in better price execution than market orders.

The following graph, which summarizes trade execution quality for iShares ETFs during May 2010, shows that orders executed at UBS, Knight, and Chicago Stock Exchange (CHX) Matching System received less favorable price execution relative to their peers. CHX and NFS executed market orders (red diamonds) at better prices than marketable limit orders (blue diamonds), but the opposite was the case at UBS and Knight. Direct Edge X exhibited the most negative WAPI among market orders.

The following graph for ProShares ETFs shows that Knight Equity Markets executed marketable limit orders with a reasonable WAPI but achieved relatively poor execution quality with market orders. Conversely, Knight Capital Markets achieved reasonable WAPI for both types of orders. Among peers, Oppenheimer exhibited the best WAPI for marketable limit orders but performed poorly with market orders. Again, one can observe how order type matters depending on the venue.

Focus: SPDR ETFs

The graphs above summarize trade execution quality for different market centers across all ETFs within a single family. Conversely, the following graph summarizes trade execution for different ETFs in the SPDR complex across all market centers.

Two ETFs with the least favorable WAPI, XSD and XBI, exhibited worse price execution for market orders than for marketable limit orders. Otherwise, market orders received more favorable execution, based on price, than marketable limit orders for most of the other tickers (even though volume for market orders was substantially lower than for marketable limit orders). In comparison to other tickers, MDY and GWX exhibited less favorable price execution for marketable limit orders. These four ETFs merit more detailed analysis.

The following graph drills down into trade execution quality by order type and order size for XSD.

Interestingly, two specific categories of market orders account for the unfavorable WAPI: (1) order quantity 2000-4999 (order size code "C") executed at UBS Securities and (2) order quantity under 500 shares (order size code "A") executed at Knight Capital Markets. Furthermore, the most favorable WAPI for market orders in this graph occurred at Knight Capital Markets for order quantity over 5000 shares (order size code "D"). Given that the fragmentation of market structure generally favors smaller orders for achieving better execution quality, the results at Knight are unexpected.

Does the same graph for XBI result in similar observations?

Yes and no. As was the case for large-quantity market orders for XSD executed at Knight, the largest-quantity market orders (2000-4999 shares) for XBI executed at NYSE Arca achieved the most favorable WAPI, certainly better than smaller-quantity market orders. At Knight, larger market orders (2000-4999 shares) received substantially less favorable WAPI than market orders with the largest-quantity (over 5000 shares). However, at Automated Trading Desk, the largest-quantity market orders (over 5000 shares) received substantially unfavorable WAPI than all other comparable orders on this graph. (Note that the total volume on some of these categories of market orders is very small.)

In the two graphs above, the WAPI for marketable limit orders was very consistent relative to market orders. Two SPDR ETFs which exhibited unfavorable execution, based on WAPI, for marketable limit orders are MDY and GWX. The following graph drills down into trade execution quality for MDY.

Marketable limit orders for MDY executed at Barclays Capital and Direct Edge X received increasingly less favorable WAPI as the order size increased. This pattern follows the broad expectation that smaller orders receive better price execution. To a lesser extent, the same pattern is evident at BATS Exchange and NASDAQ. However, market orders received more favorable WAPI than marketable limit orders, based on comparable order sizes.

The same graph for GWX yields less conclusive observations.

At NYSE Arca, Knight Capital Markets, and UBS Securities, marketable limit orders received better WAPI when the order size was smaller, but the opposite trend occurred at LavaFlow ECN (where volume was also lower). At UBS Securities, market orders received better WAPI when their order size was smaller, but at Knight there was no conclusive pattern to how well a market order would be executed according to its size.

Focus: iShares ETFs

Switching from the SPDR to iShares complex, the following graph summarizes trade execution for different ETFs in the iShares complex across all market centers.

Two ETFs stand out in terms of unfavorable WAPI for market orders: IWV and IWR. In addition, for marketable limit orders, EFG exhibited a less favorable WAPI than its peers. The following three graphs analyze what happened with each of these iShares ETFs.

For IWV, one could easily miss any anomalies in the trade execution quality. The red diamond at the lower left region represents substantially negative WAPI for large-quantity market orders executed at Knight Capital Markets. Smaller market orders at Knight received better execution. Other market centers exhibited reasonable results.

The results for IWR are surprisingly consistent with IWV. Large-quantity market orders executed at Knight received substantially negative WAPI, as indicated by the red diamond in the lower left region. The remaining data points are easier to view when this one negative outlier is excluded from the graph.

Across all five market centers, marketable limit orders received less favorable WAPI as the order size increased, with a few exceptions. At Knight, only the smallest-quantity market and marketable limit orders achieved positive WAPI. At NYSE Arca, market orders once again exhibited more favorable WAPI than marketable limit orders of comparable order size. While Automated Trading Desk executed market orders with more favorable WAPI as the order size decreased, the same pattern could not be observed for marketable limit orders.

Before switching focus to EFG, one may wonder whether orders for other iShares ETFs exhibited substantially negative WAPI at Knight. The following graph indicates that IWV and IWR were unique outliers, but some other ETFs also exhibited less favorable WAPI for market orders: IVW, IWP, IJT, IJH, and IJR.

For EFG, the least favorable WAPI for marketable limit orders occurred at E*Trade Capital Markets, although the magnitude is insignificant when compared to results for IWV and IWR.

At E*Trade, the largest-quantity marketable limit orders for EFG exhibited substantially unfavorable WAPI versus other market centers. Knight Capital Markets executed marketable limit orders with improving WAPI as the order size increased (again, counter to expectations based on general market structure). Even NYSE Arca and Pershing followed a similar pattern with respect to market orders (i.e. more favorable WAPI as order size increased), except when Pershing executed the largest-quantity market orders.

Focus: ProShares ETFs

The following graph summarizes trade execution for different ETFs in the ProShares complex across all market centers.

As shown in the next two graphs, both SQQQ and UPRO exhibited similar patterns to earlier examples for selected SPDR and iShares ETFs. In the case of SQQQ, market orders of quantity 2000-4999 executed at Knight Capital Markets received the least favorable WAPI, although not of significant magnitude in comparison to IWV and IWR.

For UPRO, the largest market orders executed at Knight received the most negative WAPI.

However, for ROM, market orders of the largest-quantity (5000 or more shares) exhibited the least favorable WAPI at Citadel Securities, although Knight and Automated Trading Desk also executed market orders with relatively less favorable WAPI.

Conclusions

Depending on the market center and specific ticker, trade execution quality, as measured by Weighted Average Price Improvement, varies wildly. As seen in the graphs above, robust generalizations are difficult to draw.

1. Marketable limit orders do not always receive better price improvement than market orders. Marketable limit orders are limit orders which should be executed immediately, as with market orders, because either the order bid is higher than the best offer or the order offer is lower than the best bid. Depending on the market center, market orders can receive better price improvement versus marketable limit orders of comparable order size. In other words, executing a market order may result in price improvement despite the fact that many erroneous trades involved stop loss market orders.

2. Depending on the specific ETF, market center does matter. Among the small sample of graphs above, some market centers repeatedly achieved less favorable price improvement than their peers. The Rule 605 disclosures cover only orders for which a market center was not specified by the investor. As a result, the data should reflect efforts to route orders to the market center which has the best NBBO at a given time. However, identifying the reasons why an order executes at a specific venue requires more data than available through the Rule 605 disclosures.

3. Major market centers with a large share of volume (usually synonymous with deep liquidity) do not necessarily achieve better price improvement. Regardless of trading volume, some market centers appear to execute market orders with better price improvement than for marketable limit orders. The same complex-level graphs shown above are available with market centers sorted from left to right in order of trading volume: SPDR, iShares, ProShares.

4. Typically, order size does matter. Depending on the market center and individual ETF, smaller orders might achieve better price improvement than larger orders, but many instances of the opposite exist. At least one should not assume that smaller orders always achieve better price improvement, especially with market orders. Since some trade execution statistics are based on small order volumes, one should study more periods of history in order to draw statistically robust conclusions.

For those seeking a more league-tablesque summary, the following graphs may be more satisfying. These graphs list the most and least favorable price improvement performances for combinations of individual ETFs and market centers. Market orders and marketable limit orders are shown separately, as their overall magnitudes of WAPI are different.

One of the market centers should be familiar by now (if not, this WSJ article might help). For those seeking a comparison, the same graphs for April are available for market orders and marketable limit orders.

Clearly, May was an exceptional month as highlighted by the price impact of ETF trades. In such challenging and volatile markets, is the quest for price improvement worthwhile, or would a market center be better off routing an order elsewhere? For investors, the dilemma is even more complicated: which ETF, order type, and order quantity would work best? There can be no assurance that if Flash Crash II ever occurs, the quality of price improvement will follow similar patterns.

Drilling Down into the Flash Crash, ETF Edition

Many media reports have commented on the severity with which stock prices collapsed during the afternoon of May 6, 2010 ("Flash Crash"). The most surprising casualty from the Flash Crash are ETF's. In general, ETF prices fluctuate consistently with the underlying index (for passive indexed, not active, portfolios) due to sufficient trading volume in the ETF itself and/or the ability for authorized dealers to use baskets of the underlying securities to profit from pricing discrepancies between the ETF and its constituent securities. ETF Trends wrote an article (referenced below) discussing the mechanisms for trading large blocks of ETF's either directly or indirectly via its constituent securities.

While ETF's justifiably raised concerns in light of the Flash Crash, the media has portrayed ETF's as a single asset class in this context. Some Wall Street Journal articles (referenced below) have presented examples of specific ETF's which suffered from extreme volatility that day. Although most of the trades which the exchanges cancelled on the evening of May 6 occurred in ETF's, many other ETF's did not incur enough price volatility to require trades to be cancelled. Few, if any, reporters have drilled down to determine which types of ETF's, and their sponsoring firms, performed better or worse on May 6.

This article attempts to reveal important distinctions among US-listed ETF's during the Flash Crash. In order to simplify the analysis, certain ETF's have been excluded. Those ETF's which utilize derivatives to achieve leverage (funds named with "ultra", "2x", "3x", and "double") are excluded in all comparisons because, by definition, their volatility is supposed to be higher than the general market, which would be confused with the exceptional volatility of un-leveraged ETF's on May 6. For similar reasons, inverse ETF's (funds named with "bear" and "short") are excluded. While the market was falling precipitously on May 6, inverse ETF's, by definition, should have not surprised investors in the same manner as long-only ETF's. Studying a single date of performance gives a limited scope of observations, but, nevertheless, readers can draw some very useful conclusions to help make future decisions on how to utilize ETF's in short-term trading or portfolio management strategies.

The simplest way to focus attention on the ETF's which exhibited unusually high volatility is to break down performance according to broad sectors. In the following table, we observe some sectors with severe price declines on May 6 regardless of high or low trading volume. The extreme magnitude of downside volatility is measured by how low prices fell in relation to the closing price that day ("Low-to-Close Difference", or "LCD"). For example, an ETF which traded as low as $1 but closed at $10 on May 6 would have an LCD of 90%. In order to better reflect the overall impact on market participants and investors, each ETF's LCD is weighted by the trading volume on May 6 to arrive at the Weighted Average Low-to-Close Difference ("WA-LCD"). Admittedly, a more accurate weighted average would consider the trading volume during the brief time during which most of the intraday volatility occurred, but that data requires more time to aggregate and process (perhaps a worthwhile objective for a future article on Fundometry).

As expected, ETF's invested in equities, including Asset Allocation, suffered from the most severe price drops. Large Cap (US) Equity ETF's performed better than Mid/Small Cap (US) Equity ETF's, which is consistent with the history of higher observed volatility in the prices of smaller companies. While the Asset Allocation sector exhibited the largest overall drop in price, that performance spans a broad range at the level of specific sectors (Moderate Allocation, Target Date, etc. as seen further below). Perhaps most significant, several sectors should be credited with avoiding the extreme price volatility of May 6: Taxable Bond, Municipal Bond, Global Bond, Currency, and Commodity ETF's. Finally, even though a broad sector may exhibit modest volatility overall, a single ETF within the sector may have performed much more poorly, as evidenced by the rightmost column "Worst Single ETF Low-to-Close Difference".

The next logical step in this analysis is to compare sectors which comprise the above broad sectors. The tiering of performance across specific asset classes and investment styles is quite apparent in the following table.

Clearly, the Small Cap Equity and Mid Cap Equity sectors rank among the worst performing sectors on the basis of the lowest trading price. Among the Small/Mid Cap group, the Small Cap Blend Equity sector held up by and far the best, with a weighted average lowest trading price 7.04% below the weighted average closing price. Industry-specific sectors suffered some excessive volatility, led by Telecom Sector Equity on relatively low volume. In order to judge the relative performance on most sectors, the Large Cap Blend Equity (which includes ETF's tracking the S&P 500) may be the benchmark. Sectors investing overseas performed relatively better. Among fixed income ETF's, the High Yield Bond sector exhibited the most downside volatility, trading 7.00% below the closing price on a weighted average basis. Relative to most sectors, the remaining bond sectors, as well as Commodity, Currency, and Target Date sectors, suffered limited downside volatility during the Flash Crash and exhibited the least negative WA-LCD figures.

Different ETF's sponsors offer products spanning various sectors. Therefore, when drilling further to the sponsoring firm level, the following tables each compare performance within a single broad sector. The term "ETF Complex" includes the ETF's offered by one firm, or within one family, which fall within the specified broad sector, and each WA-LCD is computed accordingly. The first table below compares ETF Complexes which offer Large Cap Equity portfolios.

At the ETF Complex level, trading volume shows a stronger correlation to the WA-LCD metric. ETF's from FaithShares and Grail have the lowest trading volume among their peers, perhaps to the extent that these ETF's did not trade much during the most volatie period on May 6. The leading ETF Complexes by volume - PowerShares and State Street SPDR's - performed consistently with the broader market primarily because of their high-volume products - QQQQ and SPY, respectively. (Remember, WA-LCD is weighted by trading volume on May 6.) The next most-traded ETF Complex - iShares - suffered a disappointing -44.82% weighted average drop, relative to the closing price on May 6, due to underperformance in their S&P 500 and Russell 1000 offerings. The remaining ETF Complexes which suffered disproportionate price drops also displayed very modest trading volumes relative to peers.

The comparison of ETF Complexes proceeds with the Mid/Small Cap broad sector.

Quite contrary to its performance in the Large Cap Equity broad sector, iShares exhibited the smallest overall price drop during the Flash Crash due to the high trading volume of IWM, which tracks the entire Russell 2000 index. The second-most heavily traded ETF Complex, State Street SPDR's, also held up relatively well against its peers primarily due to the high trading volume of MDY, which tracks the entire S&P MidCap 400 index. Neither of the next two most actively traded complexes - Vanguard and PowerShares - offered a specific Mid/Small Cap ETF with high volume and strong performance, resulting in significantly negative WA-LCD levels.

According the to following table, State Street SPDR's well-known family of sector ETF's, based on the S&P 500 constituents, performed respectably compared to their peers, probably due to their large share of trading volume among sector-focused equity ETF's. Van Eck's Market Vectors ETF's also suffered limited volatility, due to the outperformance of some commodity-sector equity ETF's within that complex. Likewise, the ETF Complex with the least negative WA-LCD among sector-focused products was Jefferies, with its equity portfolios based on Thomson-Reuters/Jefferies CRB indices.

On that note, the next comparison drills down into the Commodity broad sector, consisting of ETF's and ETN's which track physical commodities. Unsurprisingly, this group of ETF Complexes suffered modest declines during the Flash Crash. The PowerShares Deutsche and UBS E-TRACS complexes suffered the largest price declines, mostly due to isolated drops in specific ETF/ETN's over low volume. The most traded complexes within the Commodity broad sector, iShares and United States, exhibited consistent performance across their specific ETF's. For ETF/ETN's which give investors exposure to physical commodities, the volatility on May 6 can hardly be described as a Flash Crash.

In addition, the performance of ETF Complexes which track taxable bond portfolios exceeded the performance of their equity peers, as seen in the table below. The overall performance of PowerShares, State Street SPDR, and iShares complexes suffered from exposure to high-yield bond portfolios, specifically PHB, JNK, and HYG respectively. Otherwise, price declines in taxable bond ETF's throughout all of these complexes were modest. As should be expected, underlying ETF portfolios with longer average durations mostly exhibited greater price declines. Very few signs of a Flash Crash existed for ETF's which track taxable bond indices, and even more so for municipal bond indices.

(Similar tables for other broad sectors are available by popular demand. Interested readers may submit requests in the comments section below.)

The above comparisons clearly show that several ETF's, and some entire sectors, did not suffer from excessive volatility on May 6. Undoubtably, volatility across all market sectors was higher than normal that day. Even within complexes, the worst WA-LCD levels show some outlier downside activity in otherwise sheltered sectors. However, the market should avoid generalizations when assessing liquidity and price discovery of ETF's. Depending on its underlying portfolio, an ETF possesses a unique set of risks which may help or hurt performance during a Flash Crash scenario.

During the Flash Crash, why did some ETF's suffer less downside price risk than others, even among those which track similar indices or hold similar portfolios? The above discussion does not provide many insights which would help answer that question. Based on a yet-to-be-published analysis, no single set of factors would have predicted which ETF's outperformed on May 6. Yet to be released data covering trading activity in May might shed further light. Stay tuned.

Related articles:

How to Trade Large Blocks of ETFs Efficiently (ETF Trends)

Danger: Falling ETFs (Wall Street Journal)

A 'Flash Crash' Lesson (Wall Street Journal)