Article Index

 

Our experience with Rangeley and Vendor statements on the problem

When I first learnt of this issue from the tech press I was already aware I had a device with the affected processor, a pre-production 20TB NAS Pro box from Seagate dating back to 2014 and I was concerned if that thing would brick itself one day randomly leaving me in a conundrum to recover my data off the device and or get it repaired somehow. At this stage, knowing vaguely what brands were affected I was not that interested in the issue overall other than concern for my device.

At the Techleaders conference for Australian IT journalists in March 2017, I caught up with my local Seagate Representative to discuss the issue, but he had not heard about it at all and asked if I would send him some links on the matter. However he did understand the ramifications and suggested we could possibly swap my unit with another just as old review unit if mine fails, but that is not a foolproof fix and does not address the issues of any customers out there with retail Seagate/Lacie NAS devices.

That was the last I heard from Seagate on the Rangeley Clock Signal fault.

Not long after the original media blasts I noticed a mention of DELL in a news report and somehow I made the connection that their line of datacentre open network switches use this processor and due to the life-cycle of those products, those too were affected and DELL themselves eventually published a technical advisory on the matter.

This is was my oh $h!t moment. As a system administrator I have several of these high end Dell enterprise switches in network I manage, handling mission critical traffic. Although these are setup to be redundant and high availability, if a core switch fails, this represents a major outage on the network one way or another and as many admins know, falling back to a non-redundant system is risky. If any further outage happens in that state then everything is down, caught with pants down scenario?

Based on Dell’s advisory which was published around February 2017 I contacted both my Dell Sales and Tech Support point of contact for advise on how to go about replacing these switches as well as my PR contact for what Dell’s policy is for this issue beyond what is published in their advisory, which at the time was forward looking for a replacement program later in the year.

However, Dell Like other vendors involved were relatively quick to fix their devices in production, despite this family of switches having a long lead time. Switches made after March 2017 do not suffer from the fault, only ones made from 2014 to 2017.

Given I was personally exposed to two different brands with the fault and different approaches to customer remedies, I thought it would be prudent to reach out to other vendors we work with for media reviews to see how they are approaching the issue on an Australian or Global level.

I contacted my Australian PR contacts for Cisco, Netgear and Dell and received statements on the matter.

In March 2017, Dell Australia sent us the following statement

 

"Customer satisfaction and product quality are central to Dell’s business. A component manufactured by a supplier and included in limited Dell EMC networking products has a clock signal that can potentially degrade over time. No failures of these products have been reported to date. Beginning in July, Dell will proactively begin replacing impacted products that are under warranty or that are covered by a customer’s service contract.

 You can also access some FAQs regarding this on the support site here: http://www.dell.com/support/article/us/en/19/QNA44095/networking-clock-signal-qa?lang=EN"

There are several interesting items in this comment that in retrospect were honoured and some not, more on that later.

On The 3rd of March 2017, The Register published an executive statement from Netgear assigning blame and responsibly and defining a path for repair.

http://www.theregister.co.uk/2017/03/03/netgear_recalling_hardware_with_bad_intel_atoms/

 

"In a statement emailed to The Register, Richard Jonker, VP of SMB product line management for Netgear, said: "Netgear is taking full responsibility for this component supplier concern and we can state at this time that we are issuing a full-scale recall for the affected products. We will be contacting all registered owners of these products and providing a swap where they will receive a new product and the affected product will be returned to Netgear."

 

In the formal Knowledge Base article on the issue, Netgear actually did not recall their affected devices and as mentioned in the executive statement, offering repairs for registered users, ie users with devices in warranty, where as a formal, legally binding recall is broader reaching.

Note the statement says ‘component supplier’ ominously. During the period that this story broke, February to March 2017, Intel’s name was deliberately withheld from vendor communications on the issue. Intel did not formally disclose their processors were faulty to the media or public, apart from publishing an errata document listing the technical fixes to their products.

 

"NETGEAR will not be providing additional comment outside of the details outlined by the KB article below. 

https://kb.netgear.com/000037344/Service-Note-for-RN3130-RN3138-WC7500-and-WC7600v2?cid=wmt_netgear_organic"

Cisco was also identified by the initial reporting at the register, which I also followed up in March 2017 especially as Australia has strict consumer production laws on a federal and state level

https://www.cisco.com/c/en/us/support/web/clock-signal.html#~overview

https://www.theregister.co.uk/2017/02/03/cisco_clock_component_may_fail/

All we received from Cisco ANZ was the following just as vague statement without even a link to further information. Again the ‘supplier of anonymity’ as Intel have fixed the issue themselves relatively silently and compensated their customers for repairs behind closed doors under a confidentiality umbrella.

 

"Cisco strives to deliver technologies and services that exceed customers’ expectations, and meet rigorous quality and customer experience standards. We became aware of an issue related to a clock signal component manufactured by one supplier. We have worked with the supplier to resolve the issue, and we’re providing information and support for our customers."

 

Since Intel did not actually issue a statement of their own in 2017 on the matter I did not follow up with them and focused on their OEM partners. The issue can actually be avoided altogether depending on how the OEM implemented Intel’s chip. The typical design is affected, but some more elaborate or specific designs are not.

Intel did not say much on the matter apart from their technical documentation, however Intel's Robert Holmes Swan, CFO and executive vice president, stated at the time:

 

"But secondly, and a little bit more significant, we were observing a product quality issue in the fourth quarter with slightly higher expected failure rates under certain use and time constraints..."

 

As far as my media-centric investigation, I heard nothing further from the vendors I contacted for comment either directly or from other IT news media. After the initial round of reporting Feb to march 2017 the issue was ‘old news’

 

2018-09-21 | NVIDIA QUADRO RTX 6000 In action - Photoshop, Porsche RTX, Kubernetes, CLARA and Metropolis


2018-09-20 | NVIDIA RTX Live Demo - Ray Traced Porsche on Turing GPU


2018-09-21 | EPSON goes bright with laser projection and 4K


2018-09-15 | 8 NVIDIA TESLA V100 VOLTA GPU server from HP


2017-12-31 | Future of Networking with Extreme Networks - Eric Broockman CTO Interview


2018-02-16 | NVIDIA’S VOLTA is real, it works and it is finally here.


2018-02-07 | motorola harbourside chat and interview | moto x4, g5s plus, 360 camera & gamepad


2018-02-07 | Inwin X-Frame product tour and interview


2018-02-07 | ASUS Zenbook 3 Deluxe UX490 Ultrabook


2017-11-20 | VICHYPER - Australian Hyperloop Interview and Tour


2017-11-17 | Eugene Kaspersky & Kaspersky Lab Press Conference Sydney 2017


2017-11-17 | Eugene Kaspersky Keynote Sydney 2017


2017-05-21 | AMD RYZEN - Australian launch and Q&A Session | BIOS Update Detailed


2016-11-21 | Motorola Mobility VP Engineering Interview | Moto Z and Moto Mods Developer Kit for Makers & Modders


2016-11-30 | EPSON Tech Interview | 2016 Business and Professional Projectors


2016-11-30 | EPSON Tech Interview | 2016 Home Theatre Projectors 6000, 8000 & 9000 Series


2016-11-22 | EPSON HDR Home Theatre Projector Demo | EPSON TW9300W


2016-11-21 | HDR video Demo - AMD Radeon 'Polaris' RX480 with LG OLED TV


2016-11-30 | Intel Game Chamber 2016 Tour and Interviews | VR, Games, PC Mods, Overclocking and FUN


2016-11-30 | Intel Core i7-6950X Broadwell-E @ 5.62 GHz Live Overclock | Intel Game Chamber 2016


2016-08-29 | LG puts OLED in all the things - Integrate/CEDIA AUS 2016 Booth Tour


2016-08-12 | What is Software Defined Networking with Doug Murray, CEO Big Switch Networks.


2016-06-17 | NVIDIA DGX-1 launches in Australia - NVIDIA ANZ Interviewed


2016-05-01 | Build a kickass Dolby ATMOS Home Theatre - The Pros show us how


2016-04-01 | The Panasonic Toughbook CF-20 Story


2016-03-23 | Seagate 8TB Family, 10TB Helium & Mobile HDD 2016 Hard Disk Update


2016-03-21 | Panasonic Toughbook 20 : Fully Rugged detachable Laptop launch - Water, Drop & Weight Tested


2016-03-23 | ASUS ROG Gaming House Tour - Sydney, Australia


2016-03-01 | ASUS ROG GX700 Liquid Cooled Notebook Preview & Interview


2016-03-23 | League of Legends Legacy eSports Team Interview


2015-12-09 | Microsoft Store SYDNEY - Grand Opening Press Conference


2015-11-11 | HP 6th Gen Core Intel Skylake PC & Jet Intelligence Printing Launch - Melbourne Experience Centre


2015-11-11 | HP Experience Centre Melbourne Tour -Part 2


2015-11-11 | HP Experience Centre Melbourne Tour -Part 1


2015-08-28 | Parrot Interviewed - Flower Power Wireless Plant Monitor


2015-08-28 | Parrot Interviewed - ZiK 2.0 Headphones


2015-08-28 | Parrot Interviewed - AR. Drone, MiniDrones, Bebop Drone


2015-08-23 | MOTO G 3rd Gen IPX7 Water Proofing Demo & Moto X Style/Play/G Q&A


2015-08-21 | Motorola Mobility Interviewed - 3rd Gen Moto G, Moto X Play, Moto X Style


2015-08-20 | NVIDIA GEFORCE GTX 950 Launch Ep.1 | Gamestream Co-Op - A New way to share games


2015-08-20 | NVIDIA GEFORCE GTX 950 Launch Ep.2 | MOBA/DOTA 2 Latency and Smoothness


2015-08-08 | Intel SKYLAKE i5-6600K LN2 overclock at 6.33 GHz with ASUS Z170 ROG Hero


2015-08-07 | Intel Skylake i7-6700K @ 4.8 GHz & DDR4 @ 3.5GHz on ASUS Z170 Deluxe Overclocking Guide


2015-08-05 | ASUS Z170 Deluxe Motherboard for Intel Skylake 6th Gen Core CPU Overview


2015-08-05 | Review - ASUS Installation Tool for Intel SKYLAKE 6th Gen Core CPU


2015-08-02 | HP Pavilion and Envy 2015 Consumer Notebook First Look


2015-07-25 | HP Envy Phoenix 2015 Overclockable Gaming Tower First Look


2015-07-22 | Ashes Of The Singularity - DirectX 12 Hands-On Demo with AMD FURY X


2015-07-14 | Media Q&A with AMD and Richard Huddy - AMD Fury X ‘Fiji’ Graphics comes to Australia


2015-07-14 | MSI Interview - AMD Radeon R9 300 Gaming, Armour and Lightning GPUs