How TCP on an FPGA Looks Like

I did some research looking for existing TCP/IP FPGA Cores.  There are two basic types – 10 Gigabit and non-10 Gigabit.  There is also a range of free and commercial solutions out there, and I’m sure the usual caveats apply.  Free = no documentation or support, Commercial = super expensive.  Anyway, here is what I discovered:

The most important thing is that data comes in using an AXI4-Lite or similar interface.  What is an AXI4-Lite interface? Here is the official Xilinx documentation:

https://www.xilinx.com/products/intellectual-property/axi_lite_ipif.html

You can also simply think of it like this, on each clock cycle your FPGA reads the following signals:

For receiving data:

  • New Data is Available
  • New Data in the form of 64 bits or 8 bits

For sending data, the opposite:

  • I have new data
  • Here is the new data

No packet sizes, no lengths, nothing else.  You have 8 bytes of new data on each cycle for 10 Gigabit connections, or 1 byte of new data on each cycle for the rest.

Here is a picture to make things easier to understand:

Of course, what if you are trying to send data and the TCP Core is not ready to send data for whatever reason? Be it busy, has no space in its send buffer or does not have a valid connection/place to send the data? Well, there is a ‘Ready’ signal that you have to read before sending data as described above.

As for reading data, you have to do the complementary operation by telling the TCP Core that you are ready to read data, and then the TCP Core sends you the data.

Anyway, here is a closed-source TCP FPGA core that is non-10 Gigabit:

https://www.fpga-cores.com/cores/fc1002_mii/

And here is a commercial 10 Gigabit TCP FPGA Core from some company called Design Gateway in Bangkok, Thailand:

https://www.dgway.com/TOE10G-IP_X_E.html

In summary:

  • Each core provides 1 TCP session
  • If you want multiple sessions you have to instantiate multiple copies of the core
  • You set the parameters of your TCP connection by using registers
  • You send and receive data through a FIFO / AXI4-Lite interface

Up Next

so this looks quite simple.  I can take some data like a FIX or ITCH message, convert the date string, which in the FIX protocol is 20-something ASCII characters, convert them into a binary representation, and send out this data using a similar FIFO / AXI4-Lite interface.  Where this outgoing FIFO goes is not of concern right now, however, keep in mind that it can be any of the following:

  • Operating System/kernel TCP/IP
  • User-mode application
  • Another portion of the FPGA
  • A user-mode application running on an embedded processor which is housing the FPGA.

Zynqberry Ordered

I stumbled upon a board called the “Zynqberry” which is supposed to mimic a Raspberry Pi but with a Xilinx Zyn-7000 FPGA board present.

This board aims to be an FPGA-enabled version of the Raspberry Pi and comes with a 100MBit connector.  Unfortunately it does not support 10 Gigabit.  I mean, how could it with such a small footprint?

Anyway, this board will be perfect for me to demonstrate certain things that will make FPGA programming accessible to a hobbyist not willing to shell out thousands of dollars and to a student on a limited budget.

Here is a link to a detailed getting started guide:

https://www.digikey.com/eewiki/display/Motley/Getting+Started+with+the+ZynqBerry#:~:text=Contact-,Introduction,some%20interesting%20options%20for%20development.

And a link to where you can purchase this board yourself:

https://www.digikey.com/product-detail/en/trenz-electronic-gmbh/TE0726-03M/1686-1021-ND/6141415

My goal? Demonstrate how to solve High-Performance Computing problems using the 100Mbit connector.  Perhaps I will re-visit some of the first articles on this site and start from there.

And a link to what appears to be the official site of the manufacturer in Germany:

https://shop.trenz-electronic.de/en/Products/Trenz-Electronic/TE07XX-Zynq-SoC/TE0726-Zynq-SoC/

Possible Configurations – Starter Boards

If you are more of a hobbyist or doing this on the side, here are some possible board configurations that you can use to experiment with a 10 Gigabit connection.

Remember, a SFP+ connection is synonymous with a 10 Gigabit connection.

Zynq


ZedBoard Zynq-7000 ARM/FPGA SoC Development Board

If you are a student, you can apply for a free one through the Xilinx University Program.

FPGA: Zynq-7000 AP SoC XC7Z020-CLG484

$449.00

https://store.digilentinc.com/zedboard-zynq-7000-arm-fpga-soc-development-board/


Genesis ZU: Zynq Ultrascale+ MPSoC Development Board

Has 1 SFP+ Connector suitable for a 10 Gigabit connection.

FPGA: Zynq Ultrascale+ (XCZU3EG-SFVC784-1-E)

Pros: You can put the board on your desk and play around with other components.

Cons: You can’t plug it in to your system in a regular port. Has only 1 10 Gigabit port.  However, there is a FMC connector which you can use to add other peripherals, including a board with an SFP+ connector.

$1,149.00

https://store.digilentinc.com/genesys-zu-zynq-ultrascale-mpsoc-development-board/

 


Virtex-7


Xilinx Virtex-7 FPGA VC709 Connectivity Kit

FPGA: Virtex-7 XC7VX690T-2FFG1761C

Pros:

Plugs in to your Development Workspace

Has 4 SFP+ Connectors

Has an FMC Connector for expansion

Professional Support from Xilinx

Cons:

Pricey

Not good for a hobbyist

$4,995.00

https://www.xilinx.com/products/boards-and-kits/dk-v7-vc709-g.html


NetFPGA-SUME Virtex-7 FPGA Development Board

FPGA: Virtex-7 XC7VX690T-FFG1761-3

Pros:

Access to a big open source community

Cons:

Sometimes open source is more of a curse than a blessing.

$6,995.00

https://store.digilentinc.com/netfpga-sume-virtex-7-fpga-development-board/


Kintex-7


Genesys 2 Kintex-7 FPGA Development Board

FPGA: Kinex-7 XC7K325T-2FFG900C

Pros:

Has an FMC connector

Cons:

No out of the box support for 10 Gigabit

$999.00

https://store.digilentinc.com/genesys-2-kintex-7-fpga-development-board/


NetFPGA-1G-CML Kintex-7 FPGA Development Board

FPGA: Kintex-7 XC7K325T-1FFG676

Pros:

Cheaper than a 10 Gigabit solution.

Use to test out your FPGA based network solution before porting to 10 Gigabit

Cons:

No 10 Gigabit support – only 1 Gigabit

$1,499.00

https://store.digilentinc.com/netfpga-1g-cml-kintex-7-fpga-development-board/


 

LabVIEW FPGA Everywhere

Now you can run LabVIEW FPGA on the hardware of your choice with the new “LabVIEW FPGA IP Export Utility”. 

That sounds great, but what’s that got to do with me? (quoting Rambo 3)

Step 1 – Pick a non-National Instruments FPGA Board

You will have to pick a non-National Instruments FPGA board that has an FPGA that uses Vivado (7 Series devices and above) and has a corresponding LabVIEW FPGA board that uses a device of the same family.  What does same family mean? Well, a Virtex-7 chip such as the VX485T has the same family as the VX690T.

If you want to run LabVIEW on other hardware, you will have to pay for and upgrade to the premium version of this tool which exports to VHDL source code.  (This does not appear to be available yet, as per the documentation they request that you email them.  As for me? I am happy with the encrypted netlist for now, because it keeps things simpler and will allow for easier licensing.)

For this post, I am going to select the NetFPGA-1G-CML Kinex-7 FPGA Development Board, which has a Xilinx Kintex-7 XC7K325T FPGA.  See:

https://netfpga.org/site/#/systems/2netfpga-1g-cml/details/

Retails for $1,499.00

https://store.digilentinc.com/netfpga-sume-virtex-7-fpga-development-board/

Step 2 – Pick a Corresponding National Instruments Board

Now that you have selected the Xilinx FPGA Family that you want to use, now you have to select a National Instruments board that uses the same FPGA family, which means Kintex-7.

This page has a list of Xilinx FPGA’s and corresponding National Instruments Boards:

https://www.ni.com/en-us/support/documentation/supplemental/18/xilinx-fpga-chips-for-national-instruments-rio-devices.html

I will be selecting the PXIe-7868R board.

Step 3 – Create a LabVIEW Project using this board

Create a project with the PXIe-7858R board as one of its targets, and make a very simple top-level vi.  Here is a very simple VI Snippet:

Step 4 – Create a Build Specification

Right-click on the top-level vi and select ‘Create Build Specification’

And after you should see:

 

Step 5 – Create a Netlist Export

A new option is available under the right-click menu of the build specification called “Export to Netlist”

It runs the first steps that it normally does when you synthesize an FPGA project:

 

Be careful, the window just disappears and expects you to know where the netlist will be saved. (Hey, that’s much better than them sitting on it internally and not releasing it.  And, this is one of those RTFM issues anyway…)

C:\NIFPGA\compilation\ipexport-simple_FPGATarget_<random text>

In my case, there is a Vivado Checkpoint file and a wrapper VHD file.

The code looks pretty cool.  And remember the most important thing – ACTIVE HIGH!

NiFpgaIPWrapper_pxie7868_dash_Top_dash_Level.vhd

-- VHDL wrapper for NiFpgaAG_pxie7868_dash_Top_dash_Level
-- Generated by LabVIEW FPGA IP Export Utility
--
-- Ports:
-- reset : Reset port. Minimum assertion length: 1 base clock cycles.
-- Minimum de-assertion length: 40 base clock cycles.
-- enable_in : Enable in port. Minimum re-initialization length: 7 base clock cycles.
-- enable_out : Enable out port.
-- enable_clr : Enable clear port.
-- ctrlind_00_B : Top level control "B", sync to Clk40, u32
-- ctrlind_01_A : Top level control "A", sync to Clk40, u32
-- ctrlind_02_SUM : Top level indicator "SUM", sync to Clk40, u32
-- Clk40 : Clock "40 MHz Onboard Clock", nominal frequency 40.00 MHz, base clock

library ieee;
use ieee.std_logic_1164.all;

entity NiFpgaIPWrapper_pxie7868_dash_Top_dash_Level is
port (
reset : in std_logic;
enable_in : in std_logic;
enable_out : out std_logic;
enable_clr : in std_logic;
ctrlind_00_B : in std_logic_vector(31 downto 0);
ctrlind_01_A : in std_logic_vector(31 downto 0);
ctrlind_02_SUM : out std_logic_vector(31 downto 0);
Clk40 : in std_logic
);
end NiFpgaIPWrapper_pxie7868_dash_Top_dash_Level;

architecture vhdl_labview of NiFpgaIPWrapper_pxie7868_dash_Top_dash_Level is

component NiFpgaAG_pxie7868_dash_Top_dash_Level
port (
reset : in std_logic;
enable_in : in std_logic;
enable_out : out std_logic;
enable_clr : in std_logic;
ctrlind_00_B : in std_logic_vector(31 downto 0);
ctrlind_01_A : in std_logic_vector(31 downto 0);
ctrlind_02_SUM : out std_logic_vector(31 downto 0);
Clk40 : in std_logic;
tDiagramEnableOut : in std_logic
);
end component;

begin
MyLabVIEWIP : NiFpgaAG_pxie7868_dash_Top_dash_Level
port map(
reset => reset,
enable_in => enable_in,
enable_out => enable_out,
enable_clr => enable_clr,
ctrlind_00_B => ctrlind_00_B,
ctrlind_01_A => ctrlind_01_A,
ctrlind_02_SUM => ctrlind_02_SUM,
Clk40 => Clk40,
tDiagramEnableOut => '1'
);

end vhdl_labview;

 

Getting Started with the LabVIEW FPGA IP Export Utility

Okay, so what now?

For starters, you should know that LabVIEW 2020 FPGA uses Vivado 2019.1.

And we are going to start with the currently available version/mode – IP Export to Netlist.  (The Source Code option requires you to email them to get it unlocked)

What does this mean? It means that you can only use this utility on FPGAs that are in the Xilinx 7 Series or above.  This means:

  • Virtex-7
  • Kintex-7
  • Kintex-Ultrascale
  • Zynq-7000
  • Artix-7 (I do not know of any National Instruments boards that use this family)

See AR#53109 on Xilinx.com:

https://www.xilinx.com/support/answers/53109.html

After installing this utility, you can find the manual here:

C:\Program Files (x86)\National Instruments\LabVIEW 2020\manuals\LabVIEW FPGA IP Export Utility\FPGA_IP_Export_Getting_Started.pdf

And a working example in the ‘Example’ sub-directory:

C:\Program Files (x86)\National Instruments\LabVIEW 2020\manuals\LabVIEW FPGA IP Export Utility\Example

One important thing to remember – ACTIVE HIGH!

When you are importing this into your existing Vivado project remember that the reset and enable ports are both set to Active High.

And this is from the manual, you cannot use the following features:

  • DMA FIFO
  • Peer-to-Peer (P2P) FIFO
  • DRAM Memory
  • I/O nodes
  • Interrupt
  • User-Defined Variables

It kind of makes sense, because the features above are board specific…

Anyway, I will now look into hardware to use for the ideal trading system.  But note, with this feature, LabVIEW FPGA can be used for things such a crypto mining (if any of it is actually profitable anymore), bio-informatics, protein folding (is that the same as bio-informatics?), SETI@Home (Is this still operational?), and anything else that has an FPGA in it…

The Game Changer

Going to investigate what will be the biggest game changer for LabVIEW, FPGA’s and you.

Okay, so the National Instruments website (http://www.ni.com) is pretty hard to navigate.  I somehow saw the following under the ‘RELATED PRODUCTS’ section on some page that I have since been unable to find:

Then I was finally told by my brother to just search for it. Here is the direct link:

https://www.ni.com/en-us/support/downloads/software-products/download.labview-fpga-ip-export-utility.html#345666

Here is the online version of the readme:

http://www.ni.com/pdf/manuals/378241a.html

And please note… the description clearly says ‘export’‘third-party hardware’

This means we can take all of my strategies and knowledge for using FPGAs in LabVIEW, and apply them to any hardware. Hardware where you can write custom device drivers to optimize communications with the host processor, hardware where you can attach whatever peripherals you like… The possibilities are endless.

And to quote Bradley Cooper from “Limitless”:

“It was possible, and possible was enough”

Corona Time

So with the Coronavirus, the markets have been seeing increased volatility, and all the high frequency traders are making a lot of money, or losing a lot of money…

I scrolled through the list of repositories on the fpganow github org and found a relevant project that deals with the parsing of market data only – without any regard to the TCP/IP portion.

I will resurrect this repo and see what needs to be done in order to use it.

Now… if you are an HFT person, I recommend you look into LabVIEW FPGA, because sooner or later you will be able to run this on the platform of your choice.

https://github.com/fpganow/MarketData

What Hardware Am I Using?

I received an email from a reader asking me what controller I am using, so I figure I would make a new post with this information:

(All links open in a new tab)

And a picture (why not?):

 

More Details About TCP

So first off, I have merged my code to master, see it here:

https://github.com/fpganow/MicroBlaze_lwIP

And while I work on updating the README.md file, here are 2 slides for you to look at:

So, if you have the appropriate NI hardware, you can clone down this repository and run 1 TCP and 1 UDP session to your FPGA!

Do you have an existing FPGA solution? If you follow the Vivado Project that is exported from LabVIEW FPGA, you can probably import your existing FPGA solution to National Instruments hardware, plug in the lwip TCP/IP code that I have running on a MicroBlaze core and you will have a very rapidly customizable FPGA solution, now with TCP and UDP support.

If you already are using a TCP/IP core with your FPGA, why not give this a try? Importing cores is not that hard with LabVIEW, and if you have one why not give it a shot?

TCP is Now Working as Well!

As I was traversing my mental decision tree using a depth-first search model of digger deeper and deeper in to the lwip TCP/IP source code, I thought to myself that I should go back up the decision tree and take a deeper look at the tcpdump output.

It turns out I was setting the source and destination IP addresses to the same value. Once I set the source IP address to NULL, the TCP/IP stack auto-binded it to the IP address of the MAC and all communications started working.

Now I will do some code cleanup, will push to github and then get started on a simple demo of how this works.

After that… I will put this logic inside the FPGA and show how this can help you with your trading applications.

Code is a work-in-progress and is on the addTcp branch

https://github.com/fpganow/MicroBlaze_lwIP/tree/addTcp