How Fast Is Thunderbolt, Really? Tom's Tests With 240 TB

[razor512]

Was hoping it would be faster.


can they do the test again using a non apple computer to see how other companies implementation of thunderbolt stacks up?

(would really like to see speeds in the range of 2GB/s or so and probably a SSD based RAID 10 box that could take thunderbolt and offer good performance for workstation tasks)

 

[WINTERLORD]

I don't understand the technology because outside of a few motherboards I have not seen it in anything

 

[Fulgurant]

How fast is thunderbolt?

Not as fast as a lightning bolt.

 

[alidan]

[citation][nom]CaedenV[/nom]I am assuming you are kidding, but we absolutely need these types of connection speeds.1) Power users NEED as much speed as possible. Everyone complains that Intel's new CPUs do not improve much on performance, and AMD is sadly the usual scapegoat on that front. But the real issue is that other components are playing catchup right now, and Intel is more concerned with fighting off the horde (a beautiful horde by the way) of ARM processors creeping into Intel's desktop and server territory.Anywho, the point is that in modern systems your CPU speed is entirely second in importance to the performance to your HDD and GPU capabilities, and in more and more systems these duties are being taken over by external devices. As these more periphrial components are catching up in performance, and more and more of them are being used externally, we need connectivity capabilities that do not continue to choke them. Put another way; For what most people do (namely media consumption, office work, and internet), a late gen P4HT has more than enough processing power. Combine it with a cheap modern GPU and a SSD and most people would never know the difference between it and a brand new system. Granted, the newer systems do the same work on 10% of the power and with more integrated parts, but the processor itself is 'adequate' for what most people do. Now, if that ancient tech can still keep up with most user demands 6-8 years down the road, then what is the lifespan of modern PentiumG and i3 processors going to be? Much less bigger processors like i5 and i7s. Yes, power users will always need an upgrade, but for the masses whose most processing intensive task is watching HD video, a current gen i3 is going to have a 10+ year life of usefulness (unless something drastically changes). When you have these types of lifespans, combined with changes for the interfaces of internal components, it becomes very important to have an easy standardized highspeed external interface to ensure that these devices can use future hardware with minimal difficulty.2) People prefer external storage for a variety of reasons. Some like the 'security' of being able to keep their data with them, others like being able to work on projects in multiple locations, others are on laptops that simply do not have space for extra drives, and still others (like myself) want options to get mechanical storage into another room so that we have a truly silent working environment. In a world where gigabit Ethernet is the limit of what is available for consumer use, we need much faster external storage options.3) I know it is getting old to hear about, but the fact of the matter is that while traditional desktop computers are not dead, their death is coming. Personally I think that the desktop form factor will still be around for quite a while yet, but I will be truly surprised if my son ever has a traditional ATX desktop when he is ready for his own personal computer some 5 years from now. No doubt he will have tons of tech in his life, but I think it is going to be in the form of more specialized components. He will have a box for storage, a box for graphics power (probably integrated into the display), a box (or array of boxes) for extra processing power, and it will all be controlled by a central 'dockable' (wired or wireless) phone or tablet device which stores his personal documents and all of his software (or at least the licences for his software). Point is that as we slowly move away from PC architecture we will need high speed wired and wireless connectivity options which will tie what use to be internal devices into a mesh of external devices that may need to serve multiple users simultaneously.4) 1GB/s is not as fast as it use to be. Sure, it may take a large array of HDDs to have that kind of saturation, but it only takes 2 SSDs to hit that kind of throughput. On my own rig I have a raid0 of 1 SSDs and I get a peak throughout of 1GB/s, and an average throughput of nearly 600MB/s, and I do not have high end SSDs, these are just mid-grade Agility 3 drives. The next gen of drives coming out late this year are going to bring the read/write speed of uncompressible data much closer to the speed of compressible data, which means that drives will get a true 500+MB/s of throughput per drive no matter what type of data you throw at it. So a little box with 2 SSDs in it will be able to push 1GB/s of throughput to whatever device you hook it up to, and once again we will be at the point where connectivity is the bottleneck of the system for the foreseeable future.5) Lastly, we need something like this for an entirely different reason: Lightpeak was designed to be 'one cable to rule them all'. You were supposed to use a single daisy chain of fiber optic cable in order to connect your PC to external storage, your display, Ethernet, and even have adapters to go to things like USB or firewire devices, because Lightpeak was supposed to be a protocol agnostic connectivity standard where you could mix and match different types of devices all on one string. Obviously Thunderbolt has fallen far short of the hope that lightpeak heald, but it is still the first step towards that goal. You may not 'need' 1GB/s throughput for your HDDs... but you do need 1GB/s if you intend to run 3 displays, plus an external storage array all over a single cable. In fact, you need a lot more throughput by the time the tech catches on and we are using 4K displays and SSD arrays.[/citation]

on the p4HT side... hell no. i had one of those at 3ghz... the thing couldn't play a single hd anything at a reasonable frame rate, and if you go bluray quality, forget about it.

the internet on its own could barely work at the point the system failed and i got a quad core phenom II 955 BE

and than with people makeing their own youtube videos and editing them, unless the program offloads to the gpu and you have a good enough gpu to handle the offload, than no, the p4 is DEFINALY noticeable.

that said, putting an ssd in a computer definitely helps with responsive ness.

if you want to make a more realistic comparison, a core 2 duo to a current 6 core i7 with quad channel ram would be a good comparison... most people wouldnt notice the difference there.

the desktop as we know it wont die...
granted we may have exturnal video cards in 5 years,
but when size isnt a factor, and mobility isnt important, most people will go tower because its cheaper for better parts.

 

[beemergeek]

Based on the diagram on page 2, there is a single PCIe x4 connection between the Thunderbolt controller and the PCH controller. Since PCIe X4 scales to only 1.6GBytes/sec in throughput, your conclusions of a max of 1.4GBytes/sec aren't surprising. To go beyond that, a faster pipe (X8) would be required.

Based on 2x10Gbit connections, I'm surprised Apple didn't go with an X8 connection.

Btw, I don't agree with your intelligent RAID controller conclusion. A new dual 16GBit FC controller accessing modern storage will be far faster than a thunderbolt configuration. You'd need a x16 PCIe 3.0 (or better) slot, but that's current technology.

 

[virgult]

Finally we see the real performance numbers of USB3, by the way.
Pathetic, as expected.

 

[ElHarry]

Enterprise storage speeds.
In my test lab I get
1.1 GB/s from 10 Gb FCOE
0.75 GB/s from 8 Gb FC
1.6 GB/s from 16 Gb FC
per port, per direction.
From a quad port 16 Gb FC HBA in a Gen3 8-lane slot I get 12 GB/s (1.5 GB/s per port per direction) running write/read/compare test patterns

 

[rantoc]

[citation][nom]CaedenV[/nom]SSDs are not that expensive, and you only need 2 SSDs to hit 1GB/s of throughput. While you would need a pile of cash to get 240TB of SSDs, you only need 2 240GB SSDs to completely saturate thunderbolt, and that is entirely in the reach of just about anyone who needs that kind of throughput.[/citation]

Throughput isn't everything, latency is another factor where those 2x240 GB SSD's would pwn this insane raid0 any day. How many iops did the raid make? =)

 

[Mike Mihalik]

On Mihalik’s recommendation, we tested using a MacBook Pro with Retina Display

Nice plug, i'm sure that retina display improves the connection speed so much.

For the dual Thunderbolt ports, not the display.

 

[ddpruitt]

Curious as to what type of latencies the different configs see, particularly in RAID setups designed for redundancy.

 

[danwat1234]

USB 3.0 "Second Edition" in a year or so will have 10Gb/s speed both directions combined, so it'll be comparable to Thunderbolt.

Apple fail.

 

[sanilmahambre]

yeah! there is so much difference between 8GB/s and 12GB/s

 

[crenelle]

I'd like to see more throughput baseline tests performed with non-LaCie Thunderbolt hardware.

 

[mapesdhs]

twelve25 writes:
> I'm talking about things normal datacenters use. ...

There are far more users of Enterprise storage than just datacenters, and how many datacenters
do you know anyway?


> ... The fastest thing you'll find in a typical datacenter is 40Gb Infiniband, ...

I'm sure plenty are using FibreChannel aswell.


> ... The server you linked to is PC-based, ...

ROFL!!!! I guess you don't know anything about SGIs and shared-memory systems. An SGI UV
is about as far removed from a PC as a cheese biscuit is from continental drift.


> ... a theoretical maximum of 8GB/s (real world is likely 80% of that). So speeds like you mentioned
> above are only capable in aggregate, which you could theoretically do with thunderbolt links.

The UV is a shared-memory single-system-image machine, an application can exploit ALL the
available bandwidth, I/O performance and RAM if that is possible based on how it's coded, etc.
That's why the Onyx2 could sustain 40GB/sec more than ten years ago. The setup wasn't
exactly spartan by modern standards of course (dozens and dozens of FC links), but it works,
and that was only with a 64-CPU system using the original 1st-gen NUMAlink (1.6GB/sec total
per connection) and the old IR2 gfx. The last generation of the hw from that era was the
Onyx3900 which scales to 1024 CPUs single-image with faster connections, much better gfx
(IR4), support for SATA/SAS (Onyx2 never had that), lower memory latency, better node hop
scaling, much higher memory bandwidth, etc., but no data is available on the performance of
the equivalent O3900-based Group Station because SGI stopped publishing the details after
9/11, for obvious reasons (the plentiful detail in the original PDF I referenced was removed,
replaced with a much briefer PDF, also on my site if you want to check).

Read the following carefully and remember this is from more than a decade ago: the Group Station
could load and display a 67GByte 2D sat image in the less than 2 seconds. The max resolution
supported by the sw was more than 100K pixels across (115K IIRC). We're talking performance
levels that are still 2 orders of magnitude beyond a typical PC, but that's what was possible back
then if you had a budget of several million. Btw, I have a 36-CPU Onyx3800 in my garage (pic taken
of me next to the system a few weeks ago):

http://www.sgidepot.co.uk/misc/Ian_who_built_my_computer.jpg

It was used by Sony Pictures Imageworks. One of their engineers told me they need crazy I/O speeds
these days for some of the more compelx renders they have to do. A single frame might require the
processing of more than 500GB of data. It ain't just datacenters who have to hurl big data round now.


The I/O performance possible on a modern UV is far beyond the realms of thunderbolt. AFAIK
thunderbolt isn't yet designed to scale in this kind of manner, or do you know of any product
available *now* that allows a thousand thunderbolt links to operate in parallel?

Hehe, PC-based indeed... I almost spewed my cuppa reading that one, funniest thing I've read
in ages.

Ian.

 

[mayankleoboy1]

We need more speed in storage.
But i dont think Thunderbolt is the interface of the future. Its

1. Proprietery
2. Power sucking
3.Expensive
4. Mostly limited in Crapple
5. Not that high bandwidth anyway.
6. Not what intel initially promised : Optical fibre based, with all other connections in a TB port, while you plug only one single TB in your computer. The implementation of TB is piss-poor.

The external PCIE interface that is coming this year will stomp out Thunderbolt.

 

[janetonly42]

[citation][nom]WyomingKnott[/nom]No, but at one time we accepted that we would never need more than 640k of memory. We will get there...[/citation]
Actually it was Bill Gates who said that the idiot who thinks Obama should have more power to what ever he wants. Gates actually said that.

 

[JPForums]

On Mihalik’s recommendation, we tested using a MacBook Pro with Retina Display since he had extensive experience with the two benchmarks we're using, Disk Speed Test and AJA System Test.

No problem getting advice from an engineer who works in the field.
It is great to use these guys as resources.

Our original preference, Iometer, does exist for Mac, but Mihalik has found it to be unreliable.

Did he elaborate on this? (just curious)

However, under Mac OS 10.7.5, Mihalik could get no more than six enclosures working, regardless of whether that was a 1 x 6 or 2x 3 configuration. ... With 10.8.2, we could see drives in all twelve enclosures, but only drives in eleven enclosures would mount.

It would seem this particular setup was new to him. At this point you should have at least taken a step back and tried the setup on a Windows/Linux/Unix system. His experience here is invaluable, but you are the reviewer. If you let him call the shots, even when it isn't working as expected, then this may as well be a free LaCie advertisement. As it is, you came to a conclusion about the practical throughput limits of thunderbolt with a know faulty setup. The conclusion may in fact be valid given the performance capped early. However, a simple driver change and/or different chip manufacturer made a huge difference for USB 3.0. What are the chances we'll see a followup on a different system to solidify the conclusion? It would also be useful to find out whether the setup issues are MacBook Pro exclusive or if they effect other product lines as well.

 

[Chris Schafer]

I have done better to dual attached 10Gbe on a cisco UCS node. But you are using a laptop here. I am sure the performance is limited not by the link but by the adapter. Over iSCSI to a very nice array I have gotten >16,000 Mbps sustained. But this is using a Cisco Nexus fabric and point to point 10Gbe link into the array. And writing to cache of course.

 

[Chris Schafer]

I looked it up. The Cisco converged cards are PCIe x16 and are talking about a server platform with much less limited power windows.

[citation][nom]Chris Schafer[/nom]I have done better to dual attached 10Gbe on a cisco UCS node. But you are using a laptop here. I am sure the performance is limited not by the link but by the adapter. Over iSCSI to a very nice array I have gotten >16,000 Mbps sustained. But this is using a Cisco Nexus fabric and point to point 10Gbe link into the array. And writing to cache of course.[/citation]

 

[johnnybatters]

Can I get a few of those, please? maybe.

 

[coolronz]

Feels more like firewire v2 to me... I'll wait for usb 4.0

 

[06yfz450ridr]

I always configure my servers in raid 0 haha

 

[Duckhunt]

I have to disagree with alot of the commentary. The key problem why there is a bottle neck in moores law at this point in time is that we have a huge amount of inequality in income growth.

Who is going to afford this stuff?

If we have a shrinking middle class , why bother? A rich guy ain't going to make up for the economies of scale necessary. The richest folk I know seem to be living back in the middle ages. They won't buy high tech stuff due mostly to ignorance.

You can make up all the horse sh+t you like. In the end if the majority of the customers are living hand to mouth. Good bye upgrading technology. Who here has gotten a pay rise?

The fibre ( fiber) optic cards have been around for ages. I had seen them in the universities decades ago. To be specific I was at uni 20 years ago. We had them then. The reason why they are not everywhere is simple. They wanted get the max profits they could out of existing stuff in the ground already.

The problem is we have a do nothing congress that wants austerity rather then taxing the rich to pay to fix the economy they rich broke in the first place.

 

[segio526]

Can't wait to see this test revisited when the 20Gb ports start rolling out!

 

[Spooderman]

I don't understand why everyone rips on the slideshow. For me, it's smooth and easy to read, in a great format. I have no issues with it and almost like it.