{"id":661,"date":"2020-08-11T00:58:53","date_gmt":"2020-08-11T00:58:53","guid":{"rendered":"http:\/\/fpganow.com\/?p=661"},"modified":"2020-08-18T03:35:00","modified_gmt":"2020-08-18T03:35:00","slug":"parse-fix-messages-part-1","status":"publish","type":"post","link":"https:\/\/fpganow.com\/index.php\/2020\/08\/11\/parse-fix-messages-part-1\/","title":{"rendered":"Parse FIX Messages Part 1"},"content":{"rendered":"\n<p>For those unfamiliar with the FIX protocol, see:<\/p>\n<ul>\n<li><a href=\"https:\/\/en.wikipedia.org\/wiki\/Financial_Information_eXchange\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/en.wikipedia.org\/wiki\/Financial_Information_eXchange<\/a><\/li>\n<li><a href=\"https:\/\/www.fixtrading.org\/\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/www.fixtrading.org\/<\/a><\/li>\n<\/ul>\n<p>The FIX Protocol transfers data uncompressed and in ASCII form.\u00a0 The following data types are transferred like so:<\/p>\n<h4>Integer Value<\/h4>\n<p>To send the Integer 1,423, the TCP stream would look like this:<\/p>\n<table style=\"border-collapse: collapse; width: 100%; height: 105px;\">\n<tbody>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\"><strong>Index<\/strong><\/td>\n<td style=\"width: 33.3333%; height: 35px;\"><strong>ASCII<\/strong><\/td>\n<td style=\"width: 33.3333%; height: 35px;\"><strong>Hex<\/strong><\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">0<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;1&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x31<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">1<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;4&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x34<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 33.3333%;\">2<\/td>\n<td style=\"width: 33.3333%;\">&#8216;2&#8217;<\/td>\n<td style=\"width: 33.3333%;\">0x32<\/td>\n<\/tr>\n<tr>\n<td style=\"width: 33.3333%;\">3<\/td>\n<td style=\"width: 33.3333%;\">&#8216;3&#8217;<\/td>\n<td style=\"width: 33.3333%;\">0x33<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h4>Date Time Value<\/h4>\n<p>To send the Date Time value pair &#8220;June 4th, 1998 2:58:48 PM&#8221;, the TCP stream would look like this:<\/p>\n<table style=\"border-collapse: collapse; width: 100%; height: 702px;\">\n<tbody>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">Index<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">ASCII<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">Hex<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">0<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;1&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x31<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">1<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;9&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x39<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">2<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;9&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x39<\/td>\n<\/tr>\n<tr style=\"height: 37px;\">\n<td style=\"width: 33.3333%; height: 37px;\">3<\/td>\n<td style=\"width: 33.3333%; height: 37px;\">&#8216;8&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 37px;\">0x38<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">4<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;0&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x30<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">5<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;6&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x36<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">6<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;0&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x30<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">7<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;4&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x34<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">8<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;-&#8216;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x2D<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">9<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;1&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x31<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">10<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;4&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x34<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">11<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;:&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x3A<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">12<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;5&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x35<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">13<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;8&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x38<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">14<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;:&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x3A<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">15<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;4&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x34<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">16<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&#8216;8&#8217;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x38<\/td>\n<\/tr>\n<tr style=\"height: 35px;\">\n<td style=\"width: 33.3333%; height: 35px;\">17<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">&lt;SOH&gt;<\/td>\n<td style=\"width: 33.3333%; height: 35px;\">0x1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>A very simple thing I can do is to read the incoming TCP stream and look for any integer and datetime values and replace them with a 32-bit Integer for Integer Values, and two 32-bit Integers for the Date Time Values.\u00a0 So:<\/p>\n<p><strong>Incoming FIX stream looks like this:<\/strong><\/p>\n<p>(\u0001 is how the SOH or ASCII character 1 appears in a text editor)<\/p>\n<table style=\"border-collapse: collapse; width: 100%;\">\n<tbody>\n<tr>\n<td style=\"width: 100%;\">9=154\u000152=19980604-14:58:48\u0001<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h4>Integer Value<\/h4>\n<p>Integer value does not really save much for a small number, but that is okay for now:<\/p>\n<p>Integer field in ASCII format takes up 6 bytes:<\/p>\n<p><a href=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.ASCII_.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-709\" src=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.ASCII_.png\" alt=\"\" width=\"946\" height=\"135\" srcset=\"https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.ASCII_.png 946w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.ASCII_-300x43.png 300w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.ASCII_-768x110.png 768w\" sizes=\"auto, (max-width: 946px) 100vw, 946px\" \/><\/a><\/p>\n<p>(And the text based version in case you want to cut-and-paste)<\/p>\n<table style=\"border-collapse: collapse; width: 100%; height: 72px;\">\n<tbody>\n<tr style=\"height: 72px;\">\n<td style=\"width: 689.28px; height: 72px;\">\n<p><strong>39 3D 9A 01\u00a0<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>After converting this using an FPGA, it would take up only 4 bytes:<\/p>\n<p><a href=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.BINARY.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-710\" src=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.BINARY.png\" alt=\"\" width=\"1045\" height=\"614\" srcset=\"https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.BINARY.png 1045w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.BINARY-300x176.png 300w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.BINARY-1024x602.png 1024w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Integer.Field_.BINARY-768x451.png 768w\" sizes=\"auto, (max-width: 1045px) 100vw, 1045px\" \/><\/a><\/p>\n<h4><strong>Date Time Value<\/strong><\/h4>\n<p>Now the Date Time value however has a lot of space for saving&#8230; it can go from using 21 bytes to using 12 bytes.\u00a0 Additionally, the real savings is in the time it takes to parse this on a regular processor and in the time it takes to convert it to an appropriate type in the user-mode application.<\/p>\n<p><a href=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.ASCII_.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-711\" src=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.ASCII_.png\" alt=\"\" width=\"1773\" height=\"79\" srcset=\"https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.ASCII_.png 1773w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.ASCII_-300x13.png 300w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.ASCII_-1024x46.png 1024w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.ASCII_-768x34.png 768w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.ASCII_-1536x68.png 1536w\" sizes=\"auto, (max-width: 1773px) 100vw, 1773px\" \/><\/a><\/p>\n<table style=\"border-collapse: collapse; width: 98.5503%; height: 52px;\">\n<tbody>\n<tr style=\"height: 52px;\">\n<td style=\"width: 100%; height: 52px;\">\n<p><strong>35 32 3D 01 30 E1 3C 00 02 39 B8 01<\/strong><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>The FPGA-converted value only uses 11 bytes:<\/p>\n<p><a href=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.BINARY.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-712\" src=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.BINARY.png\" alt=\"\" width=\"1671\" height=\"738\" srcset=\"https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.BINARY.png 1671w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.BINARY-300x132.png 300w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.BINARY-1024x452.png 1024w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.BINARY-768x339.png 768w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Date.Field_.BINARY-1536x678.png 1536w\" sizes=\"auto, (max-width: 1671px) 100vw, 1671px\" \/><\/a><\/p>\n<p>What does this mean? If we are using a programming language that stores Date Time values as 2 32-bit Integers, we can convert the ASCII stream directly into this format.\u00a0 If on the other hand we are using a custom data type that stores it using the UNIX epcoh time, we can convert it directly to that.<\/p>\n<p>Or, what if we don&#8217;t care about the Date and want to discard that part of the message altogether? We can do that all in the FPGA as the data is coming in.\u00a0 No need for us to even read the entire message in to memory, we can just do this as it comes in.<\/p>\n<p>So I created a LabVIEW FPGA project that reads a text file which contains the raw FIX message or messages, and passes them down to the FPGA where I parse the data and convert each field, value pair into an optimized key, value pair and to send it back up to the host.<\/p>\n<p>Here is a screenshot of a <a href=\"https:\/\/www.ni.com\/en-rs\/support\/documentation\/supplemental\/08\/labview-subvis-explained.html\" target=\"_blank\" rel=\"noopener noreferrer\"><strong>sub-VI<\/strong><\/a> that parses data coming in and converts it into an Integer Value.\u00a0 So let&#8217;s say we are reading a FIX message and want to get the Integer value of the field number:<\/p>\n<p>(for reference)<\/p>\n<p>8=FIX.&lt;Version&gt;&lt;SOH&gt; &#8230; &lt;FieldNum&gt;=&lt;FieldValue&gt;&lt;SOH&gt; &#8230;\u00a0<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"705\" height=\"380\" src=\"http:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Get.Integer.Value_.diff_.png\" alt=\"\" class=\"wp-image-698\" srcset=\"https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Get.Integer.Value_.diff_.png 705w, https:\/\/fpganow.com\/wp-content\/uploads\/2020\/08\/Get.Integer.Value_.diff_-300x162.png 300w\" sizes=\"auto, (max-width: 705px) 100vw, 705px\" \/><\/figure>\n\n\n\n<p>What does the above <strong>LabVIEW<\/strong> <strong>FPGA<\/strong> screenshot mean?<\/p>\n<p>It means that on each clock cycle, <strong>Data_in<\/strong> has an unsigned 8-bit integer representing the data coming in from the data stream, and <strong>Reset_in<\/strong> is a boolean value telling us if we should reset the current value being stored\/saved.<\/p>\n<p>As for the output, the <strong>Valid<\/strong> boolean is set once and only once and that is when the SOH character is encountered, and the Integer Value is set to the appropriate value and is held that way until <strong>Reset_in<\/strong> is set to true again.<\/p>\n<p>Now what would this look like in VHDL?\u00a0 I don&#8217;t know, but it will take me some time to implement it, and some more time to create a test bench, and even more time to create test values and cases for this.\u00a0 But with LabVIEW? I just drew it and used existing LabVIEW tools to pass data through it.\u00a0 Go LabVIEW.<\/p>\n<p>In upcoming articles, I will expand this code to handle entire FIX protocol messages, which include:<\/p>\n<h4><strong>Heartbeat<\/strong><\/h4>\n<p>8=FIX.4.1\u00019=112\u000135=0\u000149=BRKR\u000156=INVMGR\u000134=235\u000152=19980604-07:58:28\u0001112=19980604-07:58:28\u000110=157\u0001<\/p>\n<h4><strong>LogOn<\/strong><\/h4>\n<p>8=FIX.4.2\u00019=61\u000135=A\u000134=1\u000149=EXEC\u000152=20121105-23:24:06\u000156=BANZAI\u000198=0\u0001108=30\u000110=003\u0001<\/p>\n<h4><strong>Indicator of Interest (IOI)<\/strong><\/h4>\n<p>8=FIX.4.2\u00019=154\u000135=6\u000149=BRKR\u000156=INVMGR\u000134=236\u000152=19980604-07:58:48\u000123=115685\u000128=N\u000155=SPMI.MI\u000154=2\u000127=200000\u000144=10100.000000\u000125=H\u000110=159\u0001<\/p>\n<p>For those that wish to follow, find me on github at:<\/p>\n<p><a href=\"https:\/\/github.com\/fpganow\/FIXParser\" target=\"_blank\" rel=\"noopener noreferrer\"><strong>https:\/\/github.com\/fpganow\/FIXParser<\/strong><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>For those unfamiliar with the FIX protocol, see: https:\/\/en.wikipedia.org\/wiki\/Financial_Information_eXchange https:\/\/www.fixtrading.org\/ The FIX Protocol transfers data uncompressed and in ASCII form.\u00a0 The following data types are transferred like so: Integer Value To send the Integer 1,423, the TCP stream would look like this: Index ASCII Hex 0 &#8216;1&#8217; 0x31 1 &#8216;4&#8217; 0x34 2 &#8216;2&#8217; 0x32 3 &#8230; <a title=\"Parse FIX Messages Part 1\" class=\"read-more\" href=\"https:\/\/fpganow.com\/index.php\/2020\/08\/11\/parse-fix-messages-part-1\/\" aria-label=\"Read more about Parse FIX Messages Part 1\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","footnotes":""},"categories":[1],"tags":[31,32,33,34],"class_list":["post-661","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-fix","tag-fpga","tag-labview","tag-parsing","masonry-post","generate-columns","tablet-grid-50","mobile-grid-100","grid-parent","grid-50"],"_links":{"self":[{"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/posts\/661","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/comments?post=661"}],"version-history":[{"count":26,"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/posts\/661\/revisions"}],"predecessor-version":[{"id":823,"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/posts\/661\/revisions\/823"}],"wp:attachment":[{"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/media?parent=661"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/categories?post=661"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fpganow.com\/index.php\/wp-json\/wp\/v2\/tags?post=661"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}