.NET Lambda for Capital Markets: Acquiring and Processing Bloomberg Data for a Customer’s Greenfield Deep Learning Platform 

Brian McCallion Big Data, Case Study, Lambda 0 Comments

Our favorite customer asked us to create a capital markets historical database and update it using a application written by a third party. The part Bronze Drum is developing retrieves the historical data, for example, the daily last price of the S&P 500. This is simple enough. Bloomberg Enterprise offers both a SOAP interface, and another option, called RequestBuilder. In the case of SOAP, Bloomberg issues a client certificate, and requests must be signed. AWS Lambda works well in either of the SOAP or SFTP options. 

To backfill the historical data get the historical data we created and copied the requests via SFTP a Bloomberg server that has our public key. The Bloomberg server  polls the file system, and when it finds a new request file, checks that file against pending requests,  then processes the request and writes the data to a new file containing the historical data. Once we have the historical data, this data is loaded to a table, and the tables can be queried.

The solution we’ve built processes the data that is the lifeblood of a Green Field application funded by two seasoned investors with a background in trading and pension fund advisory. The first part of the platform we built is a Continuous Delivery process that begins with Machine Learning PhDs checking in the manifest of experiments, and then packages these experiments and runs tens of thousands of randomly seeded training Workloads on a Spot Instance farm fed and managed by AWS Batch.

Traditional capital markets systems favor .NET and Windows and SQL Server. In our case we have no such baggage. However the customer sought a third party offshore data review firm to check the data for anomalies. The service also provides a schema on MS SQL Server. The application includes a small .NET application that parses daily market data closing prices for equities and indexes. Such services can be useful where a portfolio of equities is part of the data set. In our case, indexes like S&P 500 are the focus. Indexes tend to be stable. For example the S&P 500 ticker probably won’t change often, the index doesn’t split, and even where the entity that owns the S&P 500 is acquired, as in the case of SP Global (owns S&P 500) acquired by the Intercontinental Exchange, tickers don’t change.

We were not thrilled that a .NET client loads the data, as it seemed to constrain us to launching a Windows instance just to load the data. I started to look at options and to think about how to repackage the little console app for Lambda. I hope .NET Core lives up to its promise. The cross platform capability of .NET Core crept up on me and I need to follow the Microsoft developer platform more closely.