The insanity of Service Agreements and Warranty Parts

[g.lewarne]

So I work as a Technical Manager for a small independent cinema - I run the shows, do the maintenance, repairs, calibrations that sort of thing.  As a small independent, we would have never been able to outright buy Digital Cinema equipment - each screen of gear is around £150,000 to £200,000 to fit out.  That's a lot of dosh. 

So, we signed up to a funding partnership called a VPF.  I wont get into it, but essentially its a finance deal that is paid off over a 10 year period, during which the equipment is covered by a NOC and essentially unlimited free replacement parts on any non consumable modules. 

Most Digital Cinema, if it is based on DLP (Barco, NEC, Kinoton and some Chritie's) is licensed from and in many cases actually manufactured by Texas Instruments.  The Essential "guts" of a DLP cinema projector are more or less identical regardless of brand.  They even use some of the same firmware modules.  Given its all licensed from TI and/or manufactured by only a handful of companies parts are expensive. Very expensive.

A typical projector has 5 to 6 individual electronic modules, most based on FPGAs, some on ARM cores, and each module has independent self-contained firmware/software packages.  The complexity of the system is really quite astounding.  Each module is on separate boards that can be individually replaced.

In the course of the 4 years we have had our gear, the projector has been virtually rebuilt with new parts at least 3 times.  One module in particular is very prone to random, weird glitchy failures and on ours has been replaced 12 times.  This module is £4000.  12 of them provided by our service provider...in 4 years.

Half the "broken" parts get recirculated as "reconditioned" but are usually faulty or DOA when supplied, so then the service provider has to source them straight form the manufacturer at their own cost.

Just this week, we have had to replace every board of any significance to nail down a random problem with the secure connection between the playback server.  Every. Single. Board.  Approx cost, who the heck knows.  All this because, as it turns out, the Projector control board firmware does not interrogate a certain module and detect errors in the same way that the playback server does when it establishes its secure connection with the projector.  It has the capability to do so, it has exactly the same connection to this particular module as the playback server, but nope, for whatever reason the firmware engineers just didn't implement the same level of checks.  And because of this, you have the projector reporting no errors, and the server reporting a somewhat cryptic error that could mean anything between it and the module in question is faulty.

Half the time, when these boards are replaced during a fault finding mission they don't even ask me to replace the old ones that turn out to be fully functional and send the "new" ones back.  We just get to keep whatever new parts they send us.  Sometimes they don't even ask for the old stuff back. It is madness and I have no comprehension whatsoever how this company stays afloat let alone makes any kind of a profit.

And its not just me, pretty much every cinema I know with comparable equipment has the same sort of issues all the time.  And most of them are under the same VPF service company.  That's thousands of screens across Europe.

If someone has any idea how this is even financially viable for them or the manufacturer I would love to know....

Do you have any stories of warranty parts madness?

[DenzilPenberthy]

The key is probably that the "£4000" board actually cost them very little to make so it's not worth their time to do  much more than just swap out boards and not bother to ship the old ones back.  Also, if they get involved in complex fault-finding they have to pay someone who can do that. Board-swappers are much cheaper! 

[g.lewarne]

The key is probably that the "£4000" board actually cost them very little to make so it's not worth their time to do  much more than just swap out boards and not bother to ship the old ones back.  Also, if they get involved in complex fault-finding they have to pay someone who can do that. Board-swappers are much cheaper! 

Sure it cost the manufacturer a few quid to make, but the Service Company BUY their stock from the manufacturer - and not at cost price either.  And then they send them on to us, the end users with no charge whatsoever including the free international shipping, both ways... usually by next day courier and air freight

[Kilrah]

The VPF certainly has some kind of manufacturer warranty on the things so they'd get the boards replaced for free at the manufacturer's cost, and if failures are so common I'm pretty sure they'll have already yelled very hard at the manufacturer/supplier and gotten extra support and/or heavy discounts. So they wouldn't give a damn about getting the old ones back or solving the situation since they've pushed the responsibility back up the chain.

[g.lewarne]

The VPF certainly has some kind of manufacturer warranty on the things so they'd get the boards replaced for free at the manufacturer's cost, and if failures are so common I'm pretty sure they'll have already yelled very hard at the manufacturer/supplier and gotten extra support and/or heavy discounts. So they wouldn't give a damn about getting the old ones back or solving the situation since they've pushed the responsibility back up the chain.

could well be, but when you take into account shipping costs (that they pay) there is still a whole lot of money being wasted.  Just seems like a really inefficient overly complex system.

[Kilrah]

Just seems like a really inefficient overly complex system.

Much more common than you'd think in the professional world. Throwing more money to fix broken things than it would have cost to develop them properly so they worked in the first place is no rarity... but if things are that way it's that it still gets compensated for end. Mainly due to the current "value" in having a fast time to market, you'll make more money by releasing a broken system early and paying a ton to make it hold together until you can fix it than if you had taken longer and someone else had beaten you to the race... Not even counting all potential licensing deals you might be able to close thanks to it.

That wouldn't work in the consumer world thanks to the competition, but in the pro world there usually aren't many alternatives so the end user often has no choice but to eat both the hassle and the cost.

[Kleinstein]

For the cinema replacement parts, the price may not be very real. After a few years, the price for once expensive FPGAs goes down a lot. They also might have a problem taking back parts, even if working. Just testing them might be to expensive. So they keep them at there customers for later use - maybe just hoping you won't order that type of replacement if you still have one in your shelf.

There is a certain risc with these service included deals. So one some less reliable systems they will not earn much money, but with others they hardly need any service and can make up for the loss.

At university we had a HP technician for a warranty repair on an UNIX workstation. He replaced the likely rather expensive motherboard, as nearly everything, including network interface (AUI) was on the board.
Later we had the same type of fault, not covered under warranty any more - it turned out to be a replaceable fuse. Likely the same as before.

[SeanB]

If the alternative to board by board fault finding takes a lot of time, which is counted as lost revenue to the chain, and thus deducted directly from the monthly royalty payment, it is cheaper to fix the problem before the next show, or the next day evening show ( which is the only one likely outside school holidays or weekends to actually make any money over the overhead costs of running the cinema) by simply shotgunning all the parts in there with new ones that most likely are known to be working from manufacturer test.

Not sending back simply means the cost per board is very low, at least lower than the cost of fixing and having a test unit to make sure it is still fixed and will work reliably for a few months afterwards.  Although the boards are full of FPGA's and ARM cores on FPGA it is a very low cost incrementally per board. Once you have recouped the cost of development, and the price paid by the cinema covers this upfront, the actual board cost will be only a few hundred dollars, so the $4k board likely is only costing $400 to the cinema chain supplier, and the manufacturer is covering the warranty cost of the boards that fail fast. I guess they keep a record of serial numbers, so can trace failure down to individual boards easily, so simply place a warranty claim for the ones changed during troubleshooting, and not returning them saves the return cost. The manufacturer probably gets enough  payment to cover the board costs, and they likely only pay $100 per board including all costs, so can easily eat 3 extra boards and not make a loss. The odd non warranty board sale means they have covered 39 returns.

[Kilrah]

It all boils down to the economics of Human labour.

Unfortunately with the complexity of current systems someone who is good enough to find the fault is rare enough and will cost enough to make a replacement cheaper even if it has associated costs like shipping.

That to me is the main reason why the entire world is going pear-shaped.

[jpanhalt]

That is the business culture today.  Maybe some productive facilities still buy equipment and pay for maintenance, but in other endeavors, everything (i.e., capital equipment) is rented with a service agreement.  It is just the cost of business.  In some businesses, the rental includes equipment, maintenance, and expendables (e.g., reagents for testing).

We see the same principle today with subscription software* -- ensuring a continuing flow of income.

John

*Did I hear someone hiccup Windows 10 while eating a "free" glazed donut?

[Thorondor]

Hardware is "cheap"; the real money-maker is in the service contracts.

[Marco]

With a 10 year lifetime you'd think they'd negotiate a redesign of the boards at some point to not be such a giant pile of buggy poorly designed crap.

[Assafl]

Just like washing machine control boards cost as much as the washing machine.

It really doesn't. But the manufacturer wants to sell you an extended warranty OR to get you to buy a new washing machine (with support) rather than fixing the machine at parts cost.

As a total cost of the agreement - sum their costs over the life of the warranty - and subtract what you paid for (both in terms of capital expenditure as well as the warranty). So why do it if it is all paid upfront? mainly for revenue recognition. Against services revenue is only recognized as the months progress (under GAAP). Hence the vendor now has a steady revenues vs. a very lumpy revenue stream (huge revenues every time a cinema buys a projector - and nulls in between). 

[jpanhalt]

Why?

I think you missed the point several of us have made.

The price you pay is x dollars per tool per widget.   If redesigning the tool and implementing it across all customers costs more than servicing old-tool, old-tool wins.

John

[Kilrah]

With a 10 year lifetime you'd think they'd negotiate a redesign of the boards at some point to not be such a giant pile of buggy poorly designed crap.

Hopefully. But again, people who are able to redesign the boards to not be a giant pile of crap (and the actual expenses of doing so) might still cost more than regularly replacing the pile of crap.
Maybe they know a new system will come within the next 5 years, so they can advertise a 10-year lfetime knowing full well that their customers will have to buy the new thing in 5 years to stay competitive, the differnece going in their pocket of course.

[Marco]

The companies who bought the projectors in the first place aren't completely powerless in negotiations, new projectors keep being bought. From an overall point of view money is almost certainly being wasted, the technicians cost money, the downtime costs money, etc. I think likely more than a redesign.

BTW I see Christie claims a 99.999% uptime ratio.

[Kilrah]

It is impossible to make any economy involving humans "waste-free", since nobody's perfect.
Current society wants to eliminate human cost, while simultaneously keeping people busy enough not to question the system. It just can't work.

[g.lewarne]

The companies who bought the projectors in the first place aren't completely powerless in negotiations, new projectors keep being bought. From an overall point of view money is almost certainly being wasted, the technicians cost money, the downtime costs money, etc. I think likely more than a redesign.

BTW I see Christie claims a 99.999% uptime ratio.

they can claim all they want, I know of several locations that are literally dumping their Christie kit in favour of barco or nec over reliability issues. and these sites aren't even on a VPF plan, they bought it outright.

[jpanhalt]

The companies who bought the projectors in the first place aren't completely powerless in negotiations, new projectors keep being bought. From an overall point of view money is almost certainly being wasted, the technicians cost money, the downtime costs money, etc. I think likely more than a redesign.

BTW I see Christie claims a 99.999% uptime ratio.

they can claim all they want, I know of several locations that are literally dumping their Christie kit in favour of barco or nec over reliability issues. and these sites aren't even on a VPF plan, they bought it outright.

Good, and I do appreciate the nostalgia.  I live in Amish country and know some people who still make tack and horse collars.  Nevertheless, you cannot ignore where business is going.  How does your cost per occupied seat compare to your competitors?  Not only this year, but for the next few years.  What will it cost you to change over?

John

[g.lewarne]

The companies who bought the projectors in the first place aren't completely powerless in negotiations, new projectors keep being bought. From an overall point of view money is almost certainly being wasted, the technicians cost money, the downtime costs money, etc. I think likely more than a redesign.

BTW I see Christie claims a 99.999% uptime ratio.

they can claim all they want, I know of several locations that are literally dumping their Christie kit in favour of barco or nec over reliability issues. and these sites aren't even on a VPF plan, they bought it outright.

Good, and I do appreciate the nostalgia.  I live in Amish country and know some people who still make tack and horse collars.  Nevertheless, you cannot ignore where business is going.  How does your cost per occupied seat compare to your competitors?  Not only this year, but for the next few years.  What will it cost you to change over?

John

I am not sure I follow whatever you are trying to say here.

The 99.99% uptime is probably correct, but that is only accounting for lost shows (due to issues).  Our site has lost 1 (just 1) show in 3 years.  We probably have a similar uptime.  Does not mean I'm not dealing with lots of issues in between shows, or even during shows that do not necessarily require a stop.

[Kilrah]

Perfect! As seen from the equipment provider all that counts is lost shows, they won't give a damn whether you have to work 24/7 to fix their shit to get there. I.e you're being exploited.

[thm_w]

This is a bit surprising, but I'm sure its worth it to get those boards back, they just may not have a process or department able to look at them.

With a 10 year lifetime you'd think they'd negotiate a redesign of the boards at some point to not be such a giant pile of buggy poorly designed crap.

Exactly, often its not even about repairing the part, but finding a design/manufacturing flaw. How will you do this if you never look at returned parts?

We want everything worth above ~$50 back to repair. Of course you spend a few minutes at most looking at the cheaper parts.
I've repaired 1,500 parts last year, and it clearly saves money over replacing that many boards (avg cost of $150).

[Halcyon]

I guess on the topic on Service Agreements -- I shits me to tears whenever I have to deal with Cisco Systems, even as a home user.
I acquired some gear (switches, firewalls etc...) which were fully functional but headed for the bin so I put them to use at home. But trying to download software updates or even simple thing like the Cisco ASDM software is near impossible unless you first sign up for an account then link your account to a valid Cisco Service Agreement. If you don't have a current contract with Cisco, too bad, so sad.

Incidentally, if anyone here has access to Cisco downloads, I'd very much appreciate the latest software for an ASA 5505 ;-)

[g.lewarne]

This is a bit surprising, but I'm sure its worth it to get those boards back, they just may not have a process or department able to look at them.

With a 10 year lifetime you'd think they'd negotiate a redesign of the boards at some point to not be such a giant pile of buggy poorly designed crap.

Exactly, often its not even about repairing the part, but finding a design/manufacturing flaw. How will you do this if you never look at returned parts?

We want everything worth above ~$50 back to repair. Of course you spend a few minutes at most looking at the cheaper parts.
I've repaired 1,500 parts last year, and it clearly saves money over replacing that many boards (avg cost of $150).

Of the parts they do want back, there are many stories of them being "refurbished", sent to someone else who has a problem, and it is evident they had, at most, a cursory test.  I know of one site that got a mortally broken playback server - it was replaced, then after a period of time that one developed a relatively minor fault.  The next replacement was the first one they sent back, same problems included

[T3sl4co1l]

It would be nice to think that, as prices go up, quality and craftsmanship goes up.

It doesn't.  Indeed, most of the time, it goes backwards.

Tim