Xilinx FPGA bitstream

[Unixon]

Does anybody know the specs for the bitstream format of Xilinx FPGAs (officially published or obtained by reverse engineering)?
In particular, I'm interested in the old Spartan-II family.

I know I don't need these specs to simply program the FPGA as I can use the freely available ISE WebPack software on a PC,
but my greater interest is in autonomous reprogramming of FPGAs by a custom software running on a SoC/uC/whatever.
Can't clearly tell the exact purpose of this approach at the moment, I'm just looking for all possibilities.

[jancumps]

I believe that that format is unpublished.
All the open source Xilinx related projects that I've checked all rely on the Xilinx propriarity toolchain to synthesise and create the bitfile.

[miguelvp]

These are relevant for the programming part:

http://www.cypress.com/?rID=75048

http://www.cypress.com/?rID=46029

Also there is a lot of information about doing this without the EZ-USB FX2/FX3 at Xilinx.
http://www.xilinx.com/support/documentation/sw_manuals/xilinx11/platform_studio/ps_c_dld_downloading_bitstreams_fpga.htm
http://www.xilinx.com/support/documentation/sw_manuals/xilinx12_1/platform_studio/ps_c_dld_fpga_configuration_overview.htm

As for bitstream a simple search "xilinx bitstream format" returned a lot of hits including this talk from Milkymist with links to more information about reverse engineering attempts.

http://lekernel.net/fpga_toolchain_talk.pdf

[Balaur]

Do you:

a) only need to configure the device with an existing bitstream

or

b) make your own bitstream?

If a), then the configuration chapter of the Spartan II Family Datasheet is very informative. It's really easy.

If b), then I assume you really have a good reason to do this and therefore, you should be able to invest significant skill and efforts. The works in the research community (especially on topics related to design validation and fault injection through emulation) may be relevant to your interests.

[Unixon]

(b) - make my own bitstream

Well, yes, I want to do a kind of research on experimental processor architectures, some of which may require dynamical reconfiguration which in turn may include self-rewriting (e.g. by a soft-core CPU) as an extreme idea. While many things can be done purely in simulation, I'm looking forward to utilize real hardware for proof-of-concept implementations and, probably, if I ever make it to this point, for practical use.

I'm planning to start with Spartan-II/3/3E families, and Spartan-II in particular at first for two reasons: (a) they are cheap enough, so I can experiment with them without having a big problem if I destroy some of them in the process and (b) they should be simpler than modern series.

[Balaur]

Good luck!

There are many interesting resources. Particularly, this thesis has a few interesting chapters.

[bingo600]

Collin Oflynn's lubloader
http://sourceforge.net/projects/lubloader/

xsvf loaders
http://www.clifford.at/libxsvf/

http://www.simtec.co.uk/products/EB675001DIP/files/playxsvf-book/book_playxsvf.chapter_generating_xsvf.html
http://dangerousprototypes.com/tag/xsvf/
http://dangerousprototypes.com/2012/02/01/jtagwhisperer-an-arduino-jtag-programing-library/



/Bingo

[miguelvp]

Collin Oflynn's lubloader
http://sourceforge.net/projects/lubloader/

xsvf loaders
http://www.clifford.at/libxsvf/

http://www.simtec.co.uk/products/EB675001DIP/files/playxsvf-book/book_playxsvf.chapter_generating_xsvf.html
http://dangerousprototypes.com/tag/xsvf/
http://dangerousprototypes.com/2012/02/01/jtagwhisperer-an-arduino-jtag-programing-library/



/Bingo

He is looking for the bitstream format more than for how to program the xilinx.

[nctnico]

Does anybody know the specs for the bitstream format of Xilinx FPGAs (officially published or obtained by reverse engineering)?
In particular, I'm interested in the old Spartan-II family.

I know I don't need these specs to simply program the FPGA as I can use the freely available ISE WebPack software on a PC,
but my greater interest is in autonomous reprogramming of FPGAs by a custom software running on a SoC/uC/whatever.
Can't clearly tell the exact purpose of this approach at the moment, I'm just looking for all possibilities.

My advise: don't go that route! Just use the serial bitstream interface. It is easy to interface with an SPI interface. In one of my designs I used an SPI flash in which I reserved an area for the FPGA bitstream. By using some extra logic chips I could let the stream from the SPI flash end up in the serial input of the FPGA. All it took to program the FPGA is reading the SPI flash as fast as possible.

I implemented JTAG programming a Xilinx FPGA (Spartan II / IIE) from an MCU over a decade ago and that project was a real nightmare. The documentation which exists is wrong so you need to do a lot of experimenting. I actually managed to destroy an FPGA through software!

[bingo600]

He is looking for the bitstream format more than for how to program the xilinx.

Ahhh 

@OP have fun solving that one    .... But why ?

/Bingo

[Unixon]

nctnico, the major obstacle is not the programming an FPGA with a prepared bitstream, but producing a correct bitstream without ISE and not even on a PC. In other words, I want to configure FPGA with my software, not a Xilinx toolchain, because it simply may not exist where I intend to produce FPGA configuration.

If an open source FPGA architecture would be available off-the-shelf I wouldn't even bother messing around with Xilinx or Altera or whatever commercial FPGAs out there.

[miguelvp]

Unixon, best of luck to you,

The thing is that as a company you have to understand that they want their customers to feel safe with their investments without worries.

That's what keeps their employees and their families alive, so go the milkymist route and ignore that corporations are more about their workforce than the corporation itself.

But anyways, take a crack at it. You'll learn regardless if you succeed or fail.

[Unixon]

The thing is that as a company you have to understand that they want their customers to feel safe with their investments without worries.

That doesn't explain all of it. Why do they need to keep specs closed? They're just selling chips anyway. Why can't they just sell chips with open specs as other companies do?

[miguelvp]

The thing is that as a company you have to understand that they want their customers to feel safe with their investments without worries.

That doesn't explain all of it. Why do they need to keep specs closed? They're just selling chips anyway. Why can't they just sell chips with open specs as other companies do?

What other companies do? not in their field and even so, it's their prerogative to do so.

If they want to provide a safe environment it's their call. You can always make your own thing and compete if you think that's the best way to proceed.

But a buyer of techonogy might go with them because there is less risk being a closed spec.

BTW I'm not really defending them, just putting their shoes in my feet.

[Unixon]

What other companies do? not in their field and even so, it's their prerogative to do so.

I mean well... if you buy an Intel CPU or a micro-controller like AVR you also obtain a number of comprehensive datasheets which explain many things up to the assembly language and machine code of that unit so you can write your compiler to program this unit. Well, we do have the avr-gcc after all. But when it comes to FPGAs for some weird reason FPGA developer/manufacturer does not allow you to have tools to program these chips. What's the big difference? Why is it so important to keep low level architecture at secret?

[mikeselectricstuff]

What other companies do? not in their field and even so, it's their prerogative to do so.

I mean well... if you buy an Intel CPU or a micro-controller like AVR you also obtain a number of comprehensive datasheets which explain many things up to the assembly language and machine code of that unit so you can write your compiler to program this unit. Well, we do have the avr-gcc after all. But when it comes to FPGAs for some weird reason FPGA developer/manufacturer does not allow you to have tools to program these chips. What's the big difference? Why is it so important to keep low level architecture at secret?

Because they'd have to document it, and it would make it harder for them to make changes. As it is, if they want to improve the silicon, they make any corresponding changes to their software and the end-users is mostly unaffected.
The vast majority of users just don't need these details, so why put in the effort to document it externally.
The process of putting a design in an FPGA is hugely more complicated than writing software, and way beyond the ability of the vast majority of end-users.
If it's done wrong, it may appear their chips are at fault, which would be a major headache. ISTR reading that a bad configuraction can actually destroy a chip due to excessive power dissipation.
And they don't want people to be able to reverse-engineer their customers' designs from the bitstream.
From the manufacturers' point of view there is simply no good reason to make it public, and plenty of reasons not to.

[nctnico]

nctnico, the major obstacle is not the programming an FPGA with a prepared bitstream, but producing a correct bitstream without ISE and not even on a PC. In other words, I want to configure FPGA with my software, not a Xilinx toolchain, because it simply may not exist where I intend to produce FPGA configuration.

If an open source FPGA architecture would be available off-the-shelf I wouldn't even bother messing around with Xilinx or Altera or whatever commercial FPGAs out there.

How about creating your own programmable logic array on top of an FPGA? It will be slow but for a proof-of-concept that shouldn't be a problem. If you start with relatively high level blocks (adders, counters, multiplexers/demultiplexers, etc) you could avoid having to deal with boolean logic implementation & optimisation.
Alternatively there is a lot of talk on this forum about Psoc devices from Cypress which have run time reconfigurable logic.

[Unixon]

How about creating your own programmable logic array on top of an FPGA?

Yes, I was thinking about that for a while and decided to use this option if I can't do anything better.

[nctnico]

There are probably papers on this subject as well but in the past I have made more or less software configurable FPGA designs. In those designs (also on Spartan II) I used a bi-directional bus (not supported in Spartan 3 and later!) and used a block ram to store the program. Each step would read data from one functional unit and write it into another functional unit.

[miguelvp]

Say I'm a manufacturer let's say Rigol for example that use Spartan 6 FPGAs in their products 3 of them in the 4000 4 channel series. Why will i choose Xilinx again if their bitstream or worse their encryption has been compromised.

Sometimes reverse engineering can have adverse consequences to the manufacturer.

The difference with a CPU is that it holds no code (and really neither does an FPGA since it programmed at boot time) but a manufacturer has all the right in the world to keep their applications secure so that their clients feel confident on the decision of selecting that manufacturer.

They provide you with the tools to program them and if someone had a better tool I bet they might comply after a license agreement or partnership of sorts after the mandatory NDA and legal obligations.

So to me I can see why it's not public domain, they do however offer plenty of information about everything we all need other than their bitstream and encryption mechanisms.

Again, just playing the devil's advocate here, I'm really not opposed to reverse engineering.

Did you look at Milkymists presentation I linked before? It actually tell you what needs to be done but no how to actually do it. Also he links parallel efforts.

[Unixon]

Say I'm a manufacturer let's say Rigol for example that use Spartan 6 FPGAs in their products 3 of them in the 4000 4 channel series. Why will i choose Xilinx again if their bitstream or worse their encryption has been compromised.

I don't know what Rigol or other manufacturers rely on, but I know even if I were to launch a product like that I wouldn't bother worrying about keeping secrets safe.
My way of thinking tells that "when it's published - it's published, just assume your secrets are already known to everyone, quickly get over it and move on."
I doubt the bitstream protection is that much worthwhile: in comparison, a CPU code can be disassembled as well and nobody really cares about that. However, manufacturers try to protect firmware, so it's somewhat similar going on there.

Security by obscurity had always been a bad way of doing things in [at least modern] cryptography, but this principle still lives in business models which is very sad.

Yes, I browsed the presentation. I think I saw it before when I began to investigate this subject.
I though that maybe someone had greater progress in this field, but it seems that this is as far as things went up until now.

[hamster_nz]

What other companies do? not in their field and even so, it's their prerogative to do so.

I mean well... if you buy an Intel CPU or a micro-controller like AVR you also obtain a number of comprehensive datasheets which explain many things up to the assembly language and machine code of that unit so you can write your compiler to program this unit. Well, we do have the avr-gcc after all. But when it comes to FPGAs for some weird reason FPGA developer/manufacturer does not allow you to have tools to program these chips. What's the big difference? Why is it so important to keep low level architecture at secret?

All vendors bend over backwards to help you design for their chips - user guides, reference guides, App notes and so on. There is more documentation than I could ever read...

However I think that the bitstream specs are not published for good reason. I don't beleive that FPGA manufactures want to the reliability and performance of their hardware product to be determined by the quality of homebrewed synthsis software. No Open Source project could put the engineering resources, time and effort in that would be required to validate the software to a level that makes it useful for any meaningful commercial development.

I wouldn't want my core datacenter swtiches to be running with bitstreams from unvalidated tools. In fact I wouldn't expect anybody to use free tools for any product where the commercial risk exceeded the cost of a vendor toolset license - and that is pretty much any real-world commercial product.

[legacy]

No Open Source project could put the engineering resources, time and effort in that would be required to validate the software to a level that makes it useful for any meaningful commercial development.

good point, i think so, too 

[Unixon]

I don't beleive that FPGA manufactures want to the reliability and performance of their hardware product to be determined by the quality of homebrewed synthsis software.

The reliability of an FPGA is only determined by the reliability of bare silicon, it has nothing to do with bitstream, firmware or whatever other software a product user may want to run on this underlying hardware. Anything that is not hardwired on the die is a software. Speaking that someone's software determines the reliability of someone's else hardware is plain wrong and restricting the software that can run on a particular hardware or even the very ways that software can be produced is weird and surreal.

No Open Source project could put the engineering resources, time and effort in that would be required to validate the software to a level that makes it useful for any meaningful commercial development.

The openness has nothing to do with the quality of a software. I would never trust a tool I can't verify myself and all commercial tools are not verifiable as they're closed, hence all commercial tools are much more untrusted no matter what resources a manufacturer had put in verifying the product, it is virtually zero as I can't verify that myself.

I wouldn't want my core datacenter swtiches to be running with bitstreams from unvalidated tools. In fact I wouldn't expect anybody to use free tools for any product where the commercial risk exceeded the cost of a vendor toolset license - and that is pretty much any real-world commercial product.

Then just don't buy products made with tools you don't like, but don't force other people to behave exactly as you do.

[hamster_nz]

The reliability of an FPGA is only determined by the reliability of bare silicon, it has nothing to do with bitstream.

Are you saying that:

1) There is no way that a badly constructed bitstream can cause harm to an FPGA
2) That a place and route process can't turn a valid logical design into a design with reliablity issues - for example failed timing for paths, metastability issues, chip-level hot spots, dodgy startup states, clock skew over reset networks, flakeyness due to carry chain length issues, voltage and temp corner cases and so on...

Then just don't buy products made with tools you don't like, but don't force other people to behave exactly as you do.

I can't force anybody to do anything - and likewise you should allow FPGA manufactures to keep their bitstreams closed if they want to.... they don't have to share the results of their R&D with you.