var https = require('https');
var request  = https.request({
    host: 'api.cerrio.com',
    path:"api/v0.1/r/auth/Sandbox/sessions",
    method:"POST"},
 function(res) {
    console.log('got callback from sessions');
 }
});

and was getting the following exceptions:


Error: socket hang up
13:49:20 web.1 | at CleartextStream. (http.js:1272:45)
13:49:20 web.1 | at CleartextStream.emit (events.js:61:17)
13:49:20 web.1 | at Array.1 (tls.js:617:22)
13:49:20 web.1 | at EventEmitter._tickCallback (node.js:126:26)


After many dead ends the answer turned out to be the path needed to start with a /, so the correct code is:

var https = require('https');
var request  = https.request({
    host: 'api.cerrio.com',
    path:"/api/v0.1/r/auth/Sandbox/sessions",
    method:"POST"},
 function(res) {
    console.log('got callback from sessions');
 }
});

Hopefully this saves someone some time with the same issue.

Real-time software systems, not unlike most software systems, quickly become complex.  Developing under pressure with multiple sources of real-time and static data, different API’s, multiple languages, and new technologies inevitably results in spaghetti code.

Building real-time software systems with traditional technologies is hard.  The problems being solved are difficult and there are many approaches that can be taken.  It is nearly impossible to keep the entire problem space in your head and know how any given change will propagate through these systems.  And once the real-time system is built it is difficult to maintain.  You are required to remember what you, or another developer, was thinking when the system was built.

Real-time software systems are in need of visualization tools.  In fact, they are in need of real-time visualization tools.  Complex problems like those posed by real-time software systems lend themselves well to visualization tools.  The human brain is an amazing organ.  But it is better at solving certain classes of problems than it is others.  Most people can only have three to four items in their working memory at any given time.  Most real-time systems have far more than three or four logical processes and data paths.  So until we figure out a way to install additional flash memory into our prefrontal cortexes, we need to find alternative solutions.

Leveraging the human visual system is a natural fit for this problem.  Humans are a visual species.  We evolved to hunt during the day when we see well and sleep at night when we can’t.  Our brains can quickly parse enormous amounts of visual data and immediately draw our attention to whatever is the most important.  We quickly recognize shapes, colors, patterns and movement without “thinking”.

So, the benefits of visualizations are many.  Visualizations allow us to leverage the massive parallel processing capabilities of our visual system.  They allow us to work with data that is represented externally from our working memory and, therefore, process more information.  Visualizations can represent far more data than we can remember and allow us to recognize information rather than recall it.  And, unlike a screen full of code, the visualization can quickly call our attention to where our attention needs to be.

With that in mind, Cerrio built its platform to be visual.  Our visual interface makes building and maintaining real-time software surprisingly simple and intuitive.  In this series of blog posts on visualizing real-time systems, I will explore some of the different ways you can visually work with your data on Cerrio.

But before I call this post done, I would like to lay out an example of a complex real-time system built on Cerrio.  BBR Trading, a Chicago-based derivatives trading firm, has built all of their trading systems on Cerrio.  One of those systems is a risk management system.  This system combines real-time data from the stock and options exchanges, real-time data from their third-party trading venues, flat files from their clearing house, and output from their pricing models among other data.  They use that data to generate a real-time view of their risk and their profit and loss.  This is an example of a solving a very complex problem using Cerrio’s generic, visual components.  Instead of writing thousands of line of code, BBR’s developers built the system that can be visualized in the picture below.  They can look at that picture and quickly see where data is flowing from/to, and any problems that may occur with the data.  Visualization creates the much simpler picture that you see below.

BBR Trading's risk management system.

The developer maintaining BBR’s risk management system can:

1. See how data is flowing through the system
2. Quickly identify where each process gets it’s data
3. Quickly identify where data is flowing upstream
4. Be visually alerted to where problems are occurring in the system
5. Use visual process control tools to address problems
6. Quickly review custom diagnostics and performance stats
7. View documentation embedded into each process

In my series of posts on visualizing real-time systems, I will explore how developers of real-time systems can benefit from these visualizations.  If you have any specific questions, just send me an email at rbojanowski@cerrio.com.

But today I figured out a new way to screw stuff up:

            int x = 5;
            x =+ 6;
            Assert.AreEqual(11,x);

At first glance you would think this test would pass. But if you are an observant person (unlike me) or take a second look at the order of the =+ operator you’ll notice I screwed up. I meant to type +=, I was shocked that that would compile, but it does. The compiler treats it as x = (+6) which when written that way seems pretty clear. It turns out that this is just the unary plus operator at work.

Just another thing to keep an eye out for when you write some code that you swear should work but doesn’t. It is nice that in the sample code above ReSharper does warn you that the 5 is not used, in the actual code I was writing you couldn’t tell from static analysis that there was a dead assignment because I was in a loop.

The BitConverter class provides functionality for converting data to and from byte[]. This is very useful for sending data outside of your program. It is the perfect thing to use in almost every situation. The one situation where you might want to think about alternatives is if the performance of the calls you are making to BitConverter really matter (and certain conditions are met).

Before you yell at me about how premature optimization is the root of all evil let me say that I couldn’t agree more. Please don’t use the following techniques until you have used a profiler on your application and determined your calls to BitConverter are actually a limiting factor in your application.

Before we get to how to improve BitConverter we need to understand what it is doing. So let’s load it up in ILSpy and look at the method ToInt32, this method takes a byte[] and returns the int that is represent by those bytes.

public unsafe static int ToInt32(byte[] value, int startIndex)
{
	if (value == null)
	{
		ThrowHelper.ThrowArgumentNullException(ExceptionArgument.value);
	}
	if ((ulong)startIndex >= (ulong)((long)value.Length))
	{
		ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument.startIndex, ExceptionResource.ArgumentOutOfRange_Index);
	}
	if (startIndex > value.Length - 4)
	{
		ThrowHelper.ThrowArgumentException(ExceptionResource.Arg_ArrayPlusOffTooSmall);
	}
	int result;
	if (startIndex % 4 == 0)
	{
		result = *(int*)(&value[startIndex]);
	}
	else
	{
		if (BitConverter.IsLittleEndian)
		{
			result = ((int)(*(&value[startIndex])) | (int)(&value[startIndex])[(IntPtr)1 / 1] << 8 | (int)(&value[startIndex])[(IntPtr)2 / 1] << 16 | (int)(&value[startIndex])[(IntPtr)3 / 1] << 24);
		}
		else
		{
			result = ((int)(*(&value[startIndex])) << 24 | (int)(&value[startIndex])[(IntPtr)1 / 1] << 16 | (int)(&value[startIndex])[(IntPtr)2 / 1] << 8 | (int)(&value[startIndex])[(IntPtr)3 / 1]);
		}
	}
	return result;
}

Oh man, that’s a lot of code what is it doing:

• A null check
• A check to make sure that your starting index is inside the bounds and is non-negative
• A check to make sure there are four bytes to read after the starting index
• Add each byte in the array into the final result by bit shifting them into place. The order in which to do it in depends on the endian-ness of the system.

If you make two key assumptions:

• Your arguments are valid
• You will always be running on a little endian system

You can speed this code up by 50%.

So why are these assumptions safe to make? All the error checking at the beginning of the function is good, I strongly believe in defensive coding, but sometimes it’s not needed.

• The null check: if you pass null with the null check you get an exception. If you pass null without a null check you still get an exception. It is just a NullRefrenceException instead of a ArgumentNullException. In almost every circumstance I agree getting the ArguemntNullException is better than the NullRefrenceException, but we have already established that this is the .001% case where we care about performance more than almost anything else.
• Check of the startingIndex against the length of the array: this is another check that simply provides a better error message instead of changing the behavior of the program. You can see that the third check will never pass (assuming a positive value for the starting index) if the second check fails.
• The last check is possibly necessary. If you have already checked, or can statically verify that your arguments will always satisfy this condition then you can remove it. It you can’t do one of these two things than maybe it is needed.

So now we’ve taken out the safety checks, that gives us some benefit. The finally way we can speed up this function is by making the assumption that the system is little endian. The .Net platform can run on lots of platforms but the vast majority of them are little endian. The three main exceptions are the Xbox, some Winows 7 Phones and possibly (I haven’t checked this but I think it is the case) running mono on linux on some hardware. While I don’t have numbers on the percent of .Net software that is developed on those 3 platforms I feel pretty confident stating that the majority of code that is written for .Net will exclusively ever run on little endian systems.

So the extreamly optimized version of the ToInt32 function is:

public unsafe static Int32 ToInt32(byte[] data, int startingIndex)
{
    fixed (byte* numRef = &(data[startingIndex]))
    {
       return *(((int*)numRef));
    }
}

So here is the fast BitConverter file. As long as you are ok with the above two constraints, and you are in one of the rare case where performance really does matter this should help you out.

I personally see a lot of “analysis” happening that is less mature than your run-of-the-mill graduate-level statistics course or even undergraduate-level signal analysis course.

He suggests that

…you need to have analysts… report to the business side — marketing, product, CFO, COO — instead of into the engineering side.

I think these are both great points. Right now there are lots of good programmers doing shoddy work on data analysis. On the other hand, I worry that by moving the role of analyzing real time data away from the engineering side without doing something to compensate you’ll end up with business analysts doing great mathematical work but doing shoddy programming work. We see this all the time in the trading world. With all due respect to the quants out there, there is an aweful lot of fragile systems built from unreadable code by well-meaning quants.

I think there are two problems that have really yet to be solved.

The first thing that is needed are tools that allow the Math PhD’s to work with real-time data at an abstraction level they are comfortable with. The leap to big data analysis was relatively easy, because that was purely an engineering problem. The leap to meaningful real-time data analysis requires not just better implementations of numerical methods, but better ways of allowing the analysts (not engineers) to examine, change and experiment with real-time data flows while the data is flowing.

Once you give analysts the power to really experiment with dataflows the way they are used to experimenting on fixed data sets, they will be able to extract the real meaning hidden in those ever-changing data.

That’s not enough if you want the business side to be able to understand and act on those conclusions. I think the missing piece of the puzzle here is bringing the kind of beautiful visualizations that have had such an impact lately in the world of big data to the world of fast data. While there are a lot of challenges here, I think ultimately it’s a solvable problem. The human visual system is fundamentally temporal. I’m confident some of the amazing data-viz folks out there will find a way to make the output of real-time analysis intuitive and immediately understandable.

Right now however, that can be challenging. It’s tough to find a good platform that will let a semi-technical user do really powerful things. There are tools that will allow pretty much anyone to do simple things with real time streams. And of course you can have a team of programmers build you a custom solution (and then change it every time the analyst wants something different from the data).

At Cerrio, we’ve been building a real time platform that will allow you set up a system that can handle complex real time streams of data and processes them in complex ways. It has the power of a custom built solution, but it can be rapidly built without of team of highly paid programmers (or consultants). And it can be easily edited by business analysts as their requirements change without programmer intervention or complicated build/test/deploy cycles.

Try it out for free as part of our beta. Or read about some of the things you can build with it in our documentation.

Also just by having more fields you are probably lowering your conversion rate and that costs you money.

Which brings me to my question, why do I need a username? Some of the reasons that come to mind are:

• So you can log in, this is a valid argument if you are not collecting my e-mail address. If you are collecting me e-mail address then please just let me use that. Some would argue there there is some extra security in having a separate username because then a hacker has to guess both your username and password. Your E-mail could in theory be more public than your username. If people chose random usernames then I could almost buy this argument. But from my experience most people choose usernames that are either identical to or very similar to their email address or name. For instance my username is almost always bwillard, or brian.willard, I don’t think those usernames are adding a lot of security.
• Serving as a unique identifier, every account needs to have a unique ID so it can be stored in the database. But as a user I don’t care what ID you use for me, just generate a GUID or use your DB’s auto increment feature. They are much more efficient at creating unique IDs than I am.
• A combination of the previous two, you do need a value to log in with, and you do need* a value that doesn’t change to store things in the DB. So a reasonable thought is that e-mail addresses can change so we don’t want to use that as your username. But my argument is that your non-changing identifier and ‘username’ to login with don’t have to be the same thing. E-mail addresses are unique** so I don’t see any reason you can’t let people log in with their E-mail address and also let them change it.
• Having a name to display on the site, instead just display my ‘real’ name. If the name you are displaying is for human consumption then a real name is a probably a better choice than a cryptic username anyways.
• It allows for a degree of anonymity, I understand wanting to be anonymous online. When I give 10 stars to National Treasure on IMDB I don’t necessarily want that to be the first thing people see about me on Google. But for cases where I care about my privacy I will just lie for my real name. Unless the site is dealing with money they aren’t going to want to see a copy of your ID.

So I might be the only one that gets annoyed at this. And I only really notice it when I don’t get my primary username and I have to spend some time thinking of a new one. But it also adds a field to your form and probably lowers your conversion rate. It also increase the amount of information you need to store in your DB.

None of these are huge problems, but they are little problems, and I don’t really see any up side to having a username that is distinct from your E-mail address. So lets stop having them.

Am I missing something? Did you have a good reason to have a username in your system? Let me know in the comments.

*You probably don’t need to assign a non-changing value for each user, but in my experience at least having one is easier than not having one.

**I know some people share E-mail address, I don’t think this changes the argument, if they share e-mail accounts, which in my mind at least is the most private of all my online accounts, they are probably going to share an account on your service as well.

One of the places we had to update our logo was on our Twitter profile. It was interesting because after we changed our logo it took hours (some where around 3) for all of our Tweets to have the new logo on them.

Surprise number 1 was that I wouldn’t have thought that the logo would have been stored with the Tweet. I would have thought that would have been normalized out.

Surprise number 2 was that Tweets must be partitioned based on id. I would have thought that they might have been partitioned by user. It seems like there is a lot of data locality to take advantage of by storing tweets by user.

Overall I think this is a great example of a problem that requires eventual consistency, there is no reason to tax the system just to get a log rolled out everywhere all at once.

Decoupling in time is the idea that services in a real-time distributed system ought to be agnostic (decoupled) to when other services are running. Like other dependencies, timing dependencies introduce *unnecessary* complexity, so they should be avoided.

The problem pertains to late-joining transformation services. In a streaming data environment, you might have chained together a dozen services, aggregating some data here, joining in some data there. Downstream data might be dependent on a large number of upstream data producers and converters. If a naïve “middle” service joins a live stream too late, then the data that already streamed by could be lost forever to the consumers down the line.

There needs to be a way for a data transformation service to get the current state of an upstream dataset before joining the live stream of changes, and then “make it right” on the downstream dataset the service produces.

The solution is something we call “republishing”, and it pops up frequently in event-driven data scenarios. Let’s say we’re trying to make a real-time aggregation service that is decoupled in time, for example aggregating trades into a net position. Our aggregator starts late, and its republishing process is:

1) Get all the trades.

2) Performing the aggregation and create a local intermediate net position results dataset.

3) Retrieve the current state of downstream net position records.

4) Determine the difference between the two datasets and then publish changes to the downstream net positions so they are equivalent to the intermediate dataset.

5) Continue to handle trade updates in real-time, now performing real-time aggregation.

The goal with republishing is to maintain downstream data integrity and at the same time provide warm redundancy. Network outages, hardware failure and software bugs happen. Services fail, and in a real-time data environment, you need to be able to bring them back up again quickly without losing the changes that occurred in the meantime. Republishing allows a service to fail, then be brought right back up again, recovering its data state and enabling the system to “knit itself back together”.

I’ve glossed over some important considerations related to republishing that are a bit outside the scope of this blog, but the primary one is data decoupling. In order to carry out republishing, a data transformation service needs to consume the state of both upstream and downstream datasets. This is much easier if the data itself is decoupled from the producer and consumer services (think databases). And for performance reasons, it’s nice if your decoupled data nodes support a dynamic, expressive subscription model.

PubSubHubbub is a web hook based pubsub protocol. To subscribe to a PuSH-enabled blog, you need to send an HTTP Post to the hub URL and give certain parameters. One key parameter is the hub.challenge. hub.challenge is a string to verify the verification and you have to echo this back in the response. FourSquare uses OAuth authentication. It allows users to access information without having to give any credentials such as username and password. Factual gives you an unique security code. You use this security code as part of your web request. Twitter allows you to use your account name and password to access their data.

Accessing data through Factual and Twitter is very simple. As a developer the main thing I care about is the data. I want to spend the smallest amount of time possible to figure out how to get into the door to retrieve the information. Yes I understand security is important. From the company’s perspective you don’t want people to corrupt or misuse your data. You also want an easy way to deny an user or application. From a developer’s prospective I go through these security measures to make sure that there is no man in the middle giving me fake data. What about a compromise?

If seeing the data is useful to me, why not just give it to me? Give me access to fake data. Have me set up a username and password. Make it easy. Let me retrieve the data using a curl command. I do not have to write code to use your API. Tease me with the data. I can see the data format. I can see if it is relevant for my project. From there I can decide if I want to commit to you. Twitter does this perfectly. In real time I can see data go across my computer terminal.

As one of these companies, you may ask yourself what do you get in return. One is you will have more developers. You will at least have more people that will try out your system. Why make a developer spend time on something they don’t care about. I for one will invest time to figure things out if I know the data is relevant to me. If I am not sure and I can not get into the door, I will move on. There are may other similar site to yours. Time is important to me. Time is important to other developers as well.

Cheesy intros aside, once in a while a programmer of the statically typed variety runs into a situation when he or she wonders whether their life choices have led them in the right direction and whether being dynamically typed might have proven to be an easier lifestyle. One such situation occurred here when we were trying to figure out how to efficiently translate between the serialized messages “on the wire” and our program constructs.

The problem was that our clients and servers were exchanging variable-length messages without having type information at compile time. (Technically, the clients do have it but the servers don’t which means it’s still problem that needs solving!) To make matters worse, a client’s view of the data might be out of alignment with the server’s. Yeah, that was a confusing way to put things just now but we’ll just work through this together. Let’s say a server was sending a Person record { BrainChipId, FirstName, LastName, AgeInMilliseconds } to a client. It would encode it using some (in our case über efficient!) binary encoding. Great! Meanwhile, the C# client has a type called Peon { BrainChipId, AgeInMilliseconds, MothersMaidenName }. Clearly they have some fields in common but they also have some out common. Yeah, I know it’s not a real phrase. English is so asymmetricalacirtemmysa! So the client and the server agree that it’s good enough since there’s an intersection of fields and wish to continue talking. This would be a perfect fit for duck typing but for its absence from C#. So some improvisation is called for. We can resort to Reflection but in performance-critical situations it would be like taking the scenic route to your destination. We can also represent each record as a dictionary keyed on field names. This would be conceptually close to some dynamically typed languages but each set/get would be a hash table access and, although, hash table lookups are O(1) we would much prefer something that’s o(1). This is where .NET’s facilities for generating dynamic types and methods come in very handy (see System.Reflection.Emit). Our messaging layer was fitted with logic to emit getters and setters for marshaling data between the serialized, “on the wire” representation and the business logic type instances. If this sounds extremely abstract, it’s probably because it is. In the context of our meager example, the client creates an instance of the Peon type, sets its values and invokes an API call on the messaging layer (doesn’t that sound impressive?) requesting a new Peon to be added to the multi-million strong list of drones. The messaging layer invokes the make-it-happen method and the magic happens. Wouldn’t that be nice? In reality, though, once the client connects to the server, the server sends it record layouts of all of its data types so the messaging layer has something to work with. So there is enough information to construct a proper message to the server from the data contained in the submitted Peon instance and it uses the dynamically-constructed getters to extract it.

Pretty much the reverse happens when the client receives a message from the server. It creates a new Peon instance and fills it with the data from the message by invoking the setters it had pre-emitted. Then it invokes a callback to let the business logic know that it “got mail”. Beautiful and efficient. We’ve concocted a cozy illusion of duck typing’ness in our statically typed C# world.

If it types like a duck but goes about its business statically then it must be road-runner typing!