03 May 2018

Fleet of the Future

Not bad in blue, huh? This parody of the fragmented state of Bay Area transit is based on an image by Stadler Rail. There should be plenty in this image to offend almost everyone!

24 February 2018

The End of CBOSS

The rosy view, from 2011
Caltrain's troubled positive train control solution, known as CBOSS, has now been completely abandoned, to be replaced by the de-facto standard freight PTC technology known as I-ETMS. That's mostly good news, since Caltrain will no longer be stranded with a globally unique PTC system. I-ETMS is being deployed by numerous other commuter rail operators in the U.S., allowing some economies of scale and standardization.

Notwithstanding, CBOSS easily rates as the most spectacular contract failure and biggest lawsuit in Caltrain's entire history, since the Peninsula Corridor Joint Powers Board was formed in 1985.

Project expenditure history, by fiscal quarter. Fluctuations in
recent quarters are unexplained, presumably related
to termination of the Parsons contract in 2017 Q2.
Gap reflects two missing quarterly reports.
The sums expended are staggering, especially when considering that just 52 route-miles are to be fitted with PTC. To date, according to the latest quarterly capital projects report, Caltrain has expended $201 million out of $240 million budgeted for the project.

The March 2018 board packet includes a new item awarding a $49.5 million contract to Wabtec to deploy I-ETMS on the peninsula rail corridor, presumably re-using some of the hardware and communications infrastructure already installed under the CBOSS contract. The "owner's cost," borne by Caltrain to cover program management and testing, has averaged $1.2 million/month over the past five years, and should stretch well into 2019 until PTC is fully deployed and activated. (Note the December 2018 statutory deadline only requires a "revenue service demonstration" over a limited portion of the corridor). Caltrain staff estimates that owner's costs will grow the I-ETMS deployment to $59.5 million, pushing the PTC project total to at least $261 million. The board packet hints at additional future program costs, beyond the $59.5 million "switching cost" from CBOSS to I-ETMS.

How much money did Caltrain waste on CBOSS?

To estimate how much money Caltrain wasted on CBOSS, we can examine the PTC project finances of other commuter rail systems deploying I-ETMS, but without the wasteful detour into research and development of globally unique alternative solutions. These PTC-related expenses are variously reported to each operator's board of directors, in press releases, or to the FRA.

OperatorCityRoute Miles EquippedVehicles EquippedPTC Cost
MetrolinkLos Angeles249112$216M
CoasterSan Diego6017$87M
ACESan Jose06$10M

A linear regression analysis on three variables (cost per route mile, cost per vehicle, and a fixed cost allowance for control facilities) for these five commuter rail I-ETMS installations reveals that equipping one route mile of track costs on average $0.36M, equipping one locomotive or cab car costs $1.0M, and the fixed cost is $21M. These are simplistic approximations, but they do give a reasonable ballpark estimate for the underlying cost of a commuter rail I-ETMS deployment.

We then apply these estimated regression factors to Caltrain. With 52 route miles and 67 vehicles, the cost of I-ETMS deployment for Caltrain, had this solution been pursued from the beginning, would have been approximately 52 x 0.36 + 67 x 1 + 21 = $107M. This tells us two things.

First, we can infer from the $59.5M switching cost to I-ETMS that 107 - 60 = approximately $50M or just one quarter of the CBOSS sunk cost (including the fiber communications backbone and a subset of the control facilities and wayside/vehicle hardware) is salvageable for I-ETMS.

Second, since the total cost of Caltrain's PTC project is expected to reach at least $261M, we can infer that Caltrain wasted 261 - 107 = approximately $150 million on the egregious failure that was CBOSS.

$150 million flushed down the toilet. Heckuva job, Caltrain!

19 January 2018

CalMod 2.0: Three Things to Watch

UPDATE, from Caltrain TIRCP funding application
  • CalMod 2.0 is now formally known as EEP or Electrification Expansion Program
  • 100% state-funded through cap and trade program (TIRCP)
  • Consists almost entirely of option buys of 96 EMU cars for $600M
  • 17 x 8-car EMU fleet planned for start of electric service (if $$ awarded)
  • No 4-car EMUs (this is super important for future off-peak service)
  • No third bike cars.  Extra money for station bike parking
  • No level boarding.  Can kicked down road
  • Broadband internet on the EMUs at start of electric service, for a cool $14M
  • Diesel bullets redeployed to SJ - Gilroy - Salinas in unspecified future project

Caltrain was recently reported to be seeking another $630 million grant from California's cap and trade program to eliminate diesel trains entirely and to increase the passenger capacity of the new EMUs a decade earlier than previously envisioned. A previous board agenda alluded to a $756 million program known as CalMod 2.0, consisting of:
  • Full conversion to 100% EMU + capacity increase ($440M)
  • Broadband ($30M)
  • Maintenance facility improvements ($36M)
  • Level boarding and platform extensions ($250M)
While the amount reported in the press doesn't match the CalMod 2.0 tally, there may be other funding sources on tap and we are probably looking at the same package of improvements. The EMU fleet expansion is an exercise of the fully priced option for 96 additional EMU cars under the existing contract with Stadler.

There will be three important issues to keep an eye on:

1) Level Boarding

Level boarding is the logical next step after electrification, and a perfect complement: where electrification reduces time in motion, level boarding reduces time at rest. Every second of trip time saved is equally valuable, which is why cutting station dwell times is enormously important.

Not all level boarding solutions are created equal, and it's not enough for the height of the platform to equal the height of the train floor. To enable dense "blended" traffic on the peninsula corridor, what Caltrain needs is unassisted level boarding where persons of reduced mobility can board without the help of a conductor across an ADA-compliant gap. That means NO bridge plates, NO exterior lifts, and NO conductor assistance.

While the new EMUs will have the ability to dock at 51" platforms, staff and consultants evidently do not agree on a path forward towards system-wide level boarding. With a nine-figure amount being contemplated for platform extensions and level boarding under CalMod 2.0, the approach and transition strategy needs to be straightened out, and soon, to avoid enormous "do over" costs. And we should not let Caltrain claim that platform extensions for 8-car trains will cost a lot: the real price tag for that is in the range of $25 million.

2) Short EMUs for Frequent Off-Peak Service

Base order (blue) and option order
(orange) show fleet composition
for 100% electric service
The sort of service that Caltrain wants to run in the future, currently being discussed in the context of a nascent business plan, will determine the specific composition of the 96-car option order, i.e. how many of what EMU car type to buy. The wrong fleet decision could very well preclude service patterns that may be deemed preferable once the business plan effort concludes, which is why CalMod 2.0 needs to be carefully considered not to overtake or conflict with the business plan effort. That being said, you don't need an army of consultants to figure out what fleet Caltrain will need.

Use case #1: during rush hour, to run a 70 minute SF Transbay - South San Jose schedule at 6 tph per direction with 20 minute turns at each end, you need (70+20)/60 * 6 * 2 = 18 trains in service, plus one extra train available at each end of the line to protect against cascading delays, or 20 trains available for service. Allowing for a couple of trains to be down for maintenance, we need 22 trains total @ 8 cars each.

Use case #2: off-peak service running at 80 minutes SF Transbay - South San Jose at 3 tph per direction with 20 minute turns at each end, you need (80+20)/60 * 3 * 2 = 10 trains in service, plus one extra train at each end, or 12 trains available for service. Throwing in another two trains down for maintenance,  we need 14 trains total. Because it's very expensive to haul around empty seats, these must be short 4-car trains.

Supporting both of these use cases within the overall size of the Stadler order (96 cars base order + 96 cars option) requires the option order to consist primarily of 4-car EMUs, as shown in the figure at right. At peak times, 4-car EMUs would operate in pairs, mixing with the rest of the 8-car subfleet. If needed in the long term, EMUs could be extended to 12 cars by coupling 8 + 4 cars.

3) Just Say No to a Third Bike Car

Bringing a bike on Caltrain is one of the finest ways to commute; your author has done it hundreds of times. The bikes-on-board community is already gearing up to pressure Caltrain into adding a third bike car to the future 8-car EMUs, deeming the two bike cars in the base order 6-car EMUs to be inadequate. The typical argument goes that any bike "bumped" is a paying customer left behind, which is a logical argument when spare capacity is available. However, with trains at standing room only peak loads (by design!) there are plenty of potential non-bike passengers left behind. They are not "bumped" in the literal sense, since they don't even show up at the train station. Here's why: when the cost of enduring a crowded train trip becomes unbearable, the invisible hand of supply and demand pushes more and more potential riders to drive instead.

Under SRO conditions, every free bike space on the train displaces a paying passenger, a sort of "reverse bumping" effect that explains quite elegantly why, for example, the Paris RER does not and should not have dedicated bike cars. Caltrain has gone quite far enough in providing free bike space on board, and should not have a third bike car in 8-car EMUs, in everyone's interest of maximizing peak passenger capacity. In the long term, bike commuters will benefit more from world-class bike parking.

25 December 2017

CBOSS Dumpster Fire Update

The CBOSS development lifecycle,
as anticipated in 2009 on this blog.
Today we are at "point of no return."
The deadly crash of an Amtrak train near Tacoma, Washington, which would likely have been prevented if a PTC (Positive Train Control) system had been in place, has renewed the discussion of the status of PTC systems in the Bay Area. Caltrain officials say everything will be OK with CBOSS, Caltrain's very own flavor of PTC. Despite those assurances, a potent brew of ingredients is mixing together.

Bonfire of Lawsuits: After a well-chronicled program failure involving delays, cost overruns, and failure to meet milestones, Caltrain terminated the CBOSS prime contractor, Parsons Transportation Group, in February 2017. PTG and Caltrain promptly sued each other, with PTG claiming wrongful termination and Caltrain seeking up to $98 million in damages. A rich trove of documents can be accessed online under San Mateo Superior Court case number 17CIV00786, and chronicles in detail everything that went wrong with the CBOSS program. With Caltrain likely to recover some damages, PTG has now sued Alstom (formerly PTG's subcontractor and the supplier of CBOSS hardware and software) for failure to deliver a working solution. One is left to wonder how this motivates Alstom to finish the CBOSS project, since delivering a working solution to Caltrain would undermine the claim that Alstom was given an impossible task.

Dying Product: The hardware and software underlying CBOSS is known as I-ITCS, a product originally developed by GE Transportation Systems Global Signalling. While a precursor known as ITCS briefly operated on Amtrak corridors in Illinois and Indiana, it is now being displaced by the de-facto standard freight PTC system known as I-ETMS, with ITCS relegated to controlling only the grade crossing functionality in these corridors. Alstom, which acquired GE Transportation Systems in 2015, is not likely to see a future in the I-ITCS product, leaving Caltrain with a globally unique hardware and software solution. This does not bode well for product support over the lifetime of CBOSS.

Looming Deadline: the deadline imposed by Congress and the Federal Railroad Administration to successfully complete a PTC revenue service demonstration is just a year away, at the end of 2018. One year is not enough to finish, and Caltrain will almost certainly blow this deadline. Will FRA grant another extension and allow Caltrain to continue operating without PTC?

Sole Source Savior: in July 2017, avionics firm Rockwell Collins' subsidiary ARINC was awarded a sole source contract to figure out what it will take to pick up the pieces and complete the CBOSS project. ARINC completed this assessment in September, and will soon (by sheer programmatic necessity, since failure is not an option) be awarded a name-their-price sole source contract to finish a minimally working version of CBOSS that passes FRA muster. With the leverage that ARINC enjoys under these circumstances, the "re-procurement" of CBOSS will likely be (1) expensive and (2) structured such that Caltrain bears all of the risk of continued failure, i.e. cost-plus-fixed-fee rather than fixed price. With the clock ticking, the re-procurement effort has already fallen behind the planned fall 2017 schedule.

Budget Crunch: To date, Caltrain has spent over $200 million (yes, one fifth of a billion dollars!) on CBOSS with nothing to show for it. All the money allocated for CBOSS is spoken for, and a lot more (several tens of millions) will be needed to finish the project. Some of that will come from damages, but it is quite likely that 2018 will bring emergency financial maneuvers to throw more good money after bad.

Descoping of Functionality: while the first 'I' in Caltrain's I-ITCS solution stands for "Interoperable," which was one of the original selling points of CBOSS, this feature is now being thrown over the transom. Interoperability requirements contributed to the scope creep that triggered a re-design of the supposedly off-the-shelf ITCS software. It didn't help that Union Pacific was (as per usual) actively non-cooperative in helping to develop an interoperable solution, leading to Caltrain throwing in the towel and spending an additional $21.7 million (from an FRA "interoperability grant," no less!) to dual-equip seven diesel consists with the I-ETMS freight PTC system for operating on the Gilroy branch owned by UPRR. How I-ETMS freight trains will be accommodated on the peninsula corridor in I-ITCS territory is a burning question, for which the range of answers includes ditching I-ITCS and replacing it with the more viable I-ETMS, following the Amtrak example.

System Integration and Testing is Hard: while Caltrain never fails to remind us that all of the components of CBOSS are physically installed on the trains and the tracks, that is the easy part. The hard part is getting everything to operate together reliably every day, and Caltrain and their shifting band of contractors are barely getting started on this most difficult phase of the development of a new and complex safety-critical system. Integration and Testing is where the best design intentions meet cold harsh reality, and all the mistakes and omissions made during the design phase become painfully apparent. While PTG claimed in court filings that they were 90% done with CBOSS when their contract was abruptly terminated, that last 10% of troubleshooting commonly takes far more than 10% of the budget or schedule.

PTC is Hard: the legal declarations from PTG managers who ran the CBOSS program (see 17CIV00786) reveal a long list of underlying factors that caused much acrimony and remain unchanged today: (1) the specifications and standards for PTC continue to evolve, triggering continued changes and penalty testing; (2) Caltrain and its in-house consultants (the so-called "owner's team") are woefully ill-equipped and uncoordinated in their approach to complex safety-critical avionics technology development; (3) the formal contractual interactions between the "owner's team" and the vendor are complicated and delay-prone; (4) working with UPRR is a huge pain in everyone's caboose; (5) the underlying systems over which CBOSS is supposed to "overlay" are kludged-together stove pipes that, incidentally, will require nearly total re-design for the electrification program; (6) testing PTC on an operating railroad requires extensive coordination that has been demonstrated to be lacking; and so on. Strike PTG and substitute ARINC.

These ingredients will produce a situation where CBOSS does less than was promised, later than planned, and for a lot more money. No crystal ball is needed to predict that CBOSS will continue to "fail forward" to a finish line somewhere beyond 2018.

26 September 2017

Thoughts on Palo Alto

There is a vigorous discussion of grade separations now underway in Palo Alto. It misses several important points:

1) Grade Separation is not one project. Trying to come up with a single, grand unifying grade separation scheme for the entire rail corridor through Palo Alto is to over-constrain the problem and to limit the range of feasible solutions. The wide geographical spacing of the four remaining grade crossings in Palo Alto leads naturally and logically to three separate and independent projects: Alma, Churchill, and Meadow/Charleston. These three projects can be and should be completely decoupled from an engineering perspective, if not from a political perspective. The underlying geometry of Palo Alto does not lend itself to a single project.

2) Creating new cross-corridor access is not grade separation. While it is understandable that the city desires to knit together neighborhoods on opposite sides of the track by creating new places ("trench caps") where people can access the other side of the corridor, this is not grade separation and should not be funded by scarce grade separation or transportation dollars. It can't be said that the city was actively divided by the rail corridor, since the rail corridor was in place decades before Palo Alto grew into a city. While everyone agrees that new cross-corridor access would improve Palo Alto, the distinction of scope between grade separation of existing crossings (today's network topology) and new cross-corridor access (tomorrow's network topology, a nice-to-have) should remain crystal clear. Muddling the project scope will muddle the discussion of funding.

3) Split-grade solutions should be studied with due diligence. When the city commissioned a grade separation study from engineering firm Mott Macdonald, the council deliberately excluded from consideration any designs where rails or roads might rise above existing grade. From the outset, this eliminated the standard solution that every other peninsula city has adopted: San Bruno, Burlingame, San Mateo, Belmont, San Carlos, Menlo Park and Sunnyvale either already have or are planning split grade separations, where the rails are raised a bit and the streets are lowered a bit. Turning a blind eye to split grade solutions, however controversial they may be, casts doubt on the entire decision making process. Without due diligence in studying a full range of grade separation solutions, the politics of assembling the necessary funding will become unnecessarily complicated.

4) Funding matters. The most expensive options are the most popular because the cost isn't yet borne by anyone. Everything is paid for with OPM or Other People's Money. If you went to a restaurant with OPM, of course you would select the Filet Mignon (or truffles, if you're vegetarian). A selection process that ignores funding is detached from reality. This also means teaching people about orders of magnitude: capturing ill-defined revenue from new uses of 45 acres of highly impaired land that the city doesn't own, even at Palo Alto prices, doesn't begin to pay for the astronomical expense of burying the tracks. Until funding is seriously factored into decision making, it's all just unicorns and rainbows.

5) County grade separation funding is always at risk. While 2016 Measure B set aside $700 million for grade separation projects, a 3/4 majority vote of the VTA board is all that it takes to re-program some or all of that funding "as circumstances warrant" towards BART, in the exceedingly likely event that the San Jose extension goes over budget. Spend it soon, or flush it into a giant sink hole in San Jose.

Failing to properly acknowledge these realities will likely leave Palo Alto's decision making process tied in knots as other cities move forward.