There are attempts being made to merge unstructured data with market data to generate trading signals. The trouble with this strategy is that we don’t have enough ‘data’ to know which part of the merged data influences the outcome of the trade. Simply putting unstructured data into various buckets and classifying them as ‘positive’ negative’ or ‘neutral’ and merging them with structured data will distort the outcome. It may not be the right model for the development of trading algorithms.
Consider the challenges:
In the coming years, over 90% of all information will be unstructured (e.g., images, videos, MP3 files, and files based on social media and Web-enabled workloads).
Now try blending the essential market data with additional data that can make a difference in trading.
Information-management.com sets the scene for what we need for developing trading algorithms.
Consider a global equity index arbitrage strategy. Market data for the equities comprising the index needs to be tied into market data on ETFs, futures and options contracts tracking the index. Information on the index’s components should be matched against macroeconomic data, fundamental financial results and commodity spot prices. Reference data must maintain the relationships between both like and underling securities. Currency prices have to be translated in real time. Relevant data needs to be extracted from news and sentiment indicators.
After you’ve pulled together all the sources of your Big Data, you must create the algorithm that interprets, analyzes and acts on it. Then, that formula must be back tested against historical data or, in a non-production environment, real-time data. Watch what signals get generated, actions taken and results produced.”
And this is just part of the challenge. For unstructured data to work with historical market data, you also need to consider language nuances such as ‘ amazon is ready for takeoff’ or ‘Facebook misses the mark…stock blows up” try this in Hindi or Chinese and you begin to appreciate the difficulties in developing just the ‘sentiments’ part of the equation.
We know how important collective sentiments and opinions can be when making investment decisions but they do take time to form. These opinions and sentiments may not be matured enough to be useful at the time of ‘blending’. Once the data is blended, it can have a very different outcome than what you were expecting.
The better model would be to use the unstructured data and develop stand-alone analytics similar to channel research which can aid the portfolio managers and traders in forming the decision and then’ timing’ the trade. These analytics should be used as an additional source of information similar to reading WSJ and talking to analysts.
The importance of social media chatters cannot be ignored anymore when making trading decisions. Since the beginning of time, traders relied on ‘whispers’ and ‘rumors’ for trading….stocks got bought on rumors and sold on news…that’s how the market worked before the introduction of electronic trading.
All of a sudden there was a race to capture and analyze market data, write algos and put servers next to exchanges for the speedy execution of trades .Whoever could shaved off few milliseconds and got to the exchange first could make money….than the technology that allowed firms with deep pockets to developed expensive trading infrastructure got cheap, really cheap…. now anyone can set up a HFS in their bedroom and do program trading…trouble is no one can make money when everyone does the same thing. Advantages of HFS and program trading are pretty much gone. They are utilities now.
In the old days, majority of news and rumors came from the trading floors, nearby coffee shops and bars.
Today, it is coming from all over the world through social media chatters and blog posts. The process involves assessing the sentiment of the content, for example, whether comments are positive or negative regarding a particular company; sector or industry. The relevance of the content, for instance whether a company is the subject of a news article or blog, who is writing the blog, how many followers chatting and retweeting the blog.
This is a frontier territory for most, but not for the technically advanced firms…they are using big data technology to sniff and capture investors, workers and consumers sentiments. All in real time from hundreds of thousands of blogs, chats from social media sites and then capturing and neatly bundling them with other trading utilities and creating real time trading signals.
Now we have come full circle. Traders again try to capture sentiments before anyone one else can. However this time, they can combine trading utilities with unstructured data and can take advantage of a whisper 10,000 miles away from an Indian village where someone just tweeted about how a clean tech company’s product is improving lives….a company that happens to be on your watch list.
Next post: will discuss tools of the trade
Big Wall Street firms are finding ways to harness the power of big data in decision making process from customer acquisitions to portfolio management. Here is a look at what SunGard is forecasting for the Wall Street.
SunGard has identified 10 trends shaping big data initiatives across all segments of the financial services industry in 2012:
- Companies require larger market data sets and deeper granularity to feed predictive models, forecasts and trading throughout the day.
- New regulatory and compliance requirements are placing greater emphasis on governance and risk reporting, driving the need for deeper and more transparent analyses across global organizations.
- Financial institutions are ramping up their enterprise risk management frameworks to help improve enterprise transparency, auditability and executive oversight of risk.
- Financial services companies are looking to leverage large amounts of consumer data across multiple service delivery channels to uncover consumer behavior patterns and increase conversion rates.
- Emerging markets like Brazil, China and India are outpacing Europe and America as significant investments are made in local and cloud-based data infrastructures.
- Advances in big data technology will help financial services firms unlock the value of data in operations to help reduce costs and discover new revenue opportunities.
- Traditional data warehouse systems will need to be re-engineered with big data technologies to handle growing volumes of information.
- Predictive credit risk models that tap into large amounts of payment data are being adopted in consumer and commercial collections practices to help prioritize collections activities.
- Mobile applications, tablets and smartphones are creating greater pressure for company networks to consume, index and integrate structured and unstructured data from a variety of sources.
- Big data initiatives are driving increased demand for algorithms to process data, and emphasizing challenges around data security and access control as well as minimizing impact on existing systems.
With the rise of electronic trading, algorithms and high-frequency trading, market data and messaging volumes, Investment firms need new tools and strategies to handle the crush of data, identify trading signals, and predict market movements to make investment decisions.
Wall Street knows how to handle structured data. Today, however, unstructured data rules the Internet. The internet is an ocean of content, swirling with documents, news, blogs, buzz, speculation and rumor.
How does a firm exploit the web's flood of unstructured data and combine with structured data to gain an edge?
Fortunately, firms like Alivia with deep financial knowledge and big data mining capabilities are working with investment managers and developing new strategies and decision making analytics to harness the promise of big data.
Talk to us and see how Alivia can help improve your performance with big data analytics and decision making tools.