The BIG software blog

Patents in the US really makes one angry!

2011-08-06T01:03:00.004+02:00

Over the last couple of years I've heard a lot of stuff around the impact of Software Patents, mainly in the US.

Google is desperate to beef up it's "war chest" of patents, as proven by the acquisition of Motorola. The runaway auction on the Nortel patent portfolio being another example. The patents are not really individually evaluated but are bought in bulk, 400k USD / patent!

Anyways, I was reffered to an extremely interesting story that ran on This american life that dug down into the issue that I highly recommend. A warning might be in place, you might feel enraged but yet powerless to do anything ... like I did.

Finally light weight virtualization for Linux..

2011-06-28T00:46:00.004+02:00

I've been quite a fan of Solaris Containers / Solaris Zones for quite some time. For a lot of use cases they provide a very good compromise between resource partitioning, fault isolation and efficiency. They are a very good fit for a system landscape with uniform OS-environments, since they are all running the same kernel, but still logically separate environments, e.g. in software as a service situations where there are multiple customer instances of the same service.

Now, I'm probably a late-comer to this but it seems that the Linux camp is closing in. I read a twitter engineering blog post which led me on a click stream to the linux containers project. through the Mesos project which twitter is apparently building it's cloud with. Anyways it seems to have been around for quite a while since the news archive goes back all the way to 2008. It's a bit more limited in scope from Solaris Zone from what I can gather but cool and basically addressing the same architectural tradeoff sweetspot as Solaris Zones.

I'll check it out in more detail when I have a more urgent itch to scratch, but in the meanwhile though I'd let you know.

The truth about social networks

2011-04-29T12:51:00.001+02:00

http://gigaom.com/2011/04/28/the-lies-that-social-networks-keep-telling-themselves/

Insiteful about social networks at gigaom which is quickly becoming my favorite among thoughtful "slow technews"

Are we all Aliens?

2011-03-05T23:33:00.002+01:00

Intriguing thought, are we all Aliens? See this article. This could of course be put in the same hoax category as "Cold fusion" in a couple of years, but they appear to have gone through some effort the vet the result and extensively peer-review.

Anonymity on the net is officially dead!

2010-10-30T00:25:00.020+02:00

Background

Your presence on the internet has always been possible to trace to some degree, at least after the introduction of cookies. A cookie is basically a mechanism by which a server application can read a value from a client.

Due to the stateless nature of the underlying protocols some mechanism of tracking your interactions with the application really is necessary, otherwise each request on the service would be considered in isolation. The most common technique is to use cookies, but there are alternatives like URL-rewriting that could serve the same purpose, here is a good introduction. The tracking cookies for these use cases doesn't need to persistent and can be expired when the browser terminates.

The Light Side

So this is the session user tracking which is generally considered quite benign and at least a necessary evil to have some amount of usability when interacting with web applications.

Now what if the cookie is stored persistently in your browser? That can allow convenient functions in the web application between sessions, like remembering your username, user preferences, entry points into the web application, customized content, or even automatic sign-in etc. Let's call this persistent user tracking.

The Dark Side

Persistent user tracking can also be used for less obvious and less beneficial effects for the user. Common usage includes add-tracking, user browsing history tracking and user behavior analysis among other things.

There are ways in which we are led to believe that we can protect ourselves from this. For example, putting the browser into a mode where cookies are accepted only into transient memory, no matter the type of cookie (persistent or temporary). Other actions thought to have an effect against this tracking is to occasionally clear out all cookies and cache entries.

End game

Now this is pretty much proven to be completely ineffective.

Proof #1 - The Evercookie

There is a combination of techniques that is collectively called the "Evercookie" that pretty much voids all hope of avoiding user tracking, at least for the average net citizen. The Evercookie is invented by Samy Kamkar and is described on his webpage, there is also a experiment lab where you can experiement with the Evercookie. The Evercookie includes the following components:

Standard HTTP Cookies - Simple enough
Local Shared Objects (Flash Cookies) - "Flashcookie", having a small flash program store a value that is cross-browser and outside of any browser "Clear data" operation
Silverlight Isolated Storage - The same thing but for silverlight
Storing cookies in RGB values of auto-generated, force-cached - Really innovative mechanism by storing a cookie value using steganography in a picture that is cached
PNGs using HTML5 Canvas tag to read pixels (cookies) back out
Storing cookies in Web History - Using browsing history in an innovative way with predefined relative links and query with javascript whether they are present in the history
Storing cookies in HTTP ETags - Pretty straight forward use of the Etag to store the cookie value
Storing cookies in Web cache
window.name caching - Storing data in the DOM-tree connected to a browser window / tab, only available as long as the browser window is open but can be used to provide the "stickyness"
Internet Explorer userData storage - IE specific technology for cookie like storage that was never standardized
HTML5 Session Storage - HTML5 versions of the technologies above
HTML5 Local Storage - HTML5 versions of the technologies above
HTML5 Global Storage - HTML5 versions of the technologies above
HTML5 Database Storage via SQLite - A full blown database available in HTML5

The nature of the beast is such that the combination of techniques is extremely sticky and difficult to get rid of. If one of the components of the evercookie is dealt with there are other components that still make it possible to track the user. If not ALL traces of the evercookie is removed simulataneously the components restablishes themselves. The fundamental concept of many of the components require javascript to be enabled for it to be effective, so this could potentially be thwarted, e.g. with the No-script firefox plugin, but a lot of modern websites really become subfunctional without javacript and not really for the average net-citizen.

Proof #2 - Panopticlick

The EFF ran an experiement on how to identify users that have thwarted explicit cookies buy using fingerprinting of the browsers, some aspects not even using javascript. They received a very good user tracking result, over 90% tracking capability.

In conclusion

For the average net citizen anonymity and not being tracked is not really feasible. So act accordingly where privacy and security is concerned. Some thechniques that might be helpful:

* Compartmentalize your browsing, e.g. using different virtual machines for different purposes

* Use the private browsing mode / incognito mode offered by your browser, eventually they may become good enough to thwart the evercookie and similar "attacks".

* Assume that you are always tracked and act accordingly

Why a private SSL certificate is better...

2010-10-03T00:37:00.012+02:00

Some SSL / TLS basics is available here. SSL provides authentication, confidentiality and integrity. Authentication of the server, and less commonly used the server can also request authentication of the client.

The confidentiality and integrity relies on pretty good theoretical grounds in hard-core cryptography so not much else needs to be said there. Most reasonable ciphers are computationally infeasible to crack.

The authentication is what really distinguishes a public key infrastructure such as SSL/TLS from conventional cryptography where security basically depends on two parties sharing a common secret. For PKI the chain of trust is critical to the integrity of the authentication process, ultimately authentication is provided by performing a "proof-of-possession of private key". Your client needs to rely on the Certificate authority that has issued the certificate for the server which you want to authenticate. Without this trust the whole PKI breaks down and you could be communicating with anybody, receiving your sensitive data.

So the title of the blog post sounds like it's contrary to security best practice, right? The common knowledge is that the certificates should be issued by reliable parties.

There are a few problems with this though. The most platforms in use have crypto packages which include "well known" CAs. The platforms have really bloated their CA-stores to include a truly astounding amount of root CAs. Windows for example have well over 200 approved CAs in their root CA-store. Some of the problems:

Not all CAs use the same high standards when verifying ownership and identity

Some CAs are under the control of governments with questionable records of privacy and due process in wiretapping
Server OSs quite often include the same"client" trust stores as client OS

With the proliferation of root CAs available the risk of a man-in-the-middle attacks are quite significant. The way it can be pulled off is:

The client doesn't specify that it only relies on a certificate issued to domain www.acme.org from a specific CA
An attacker has recieved the right to become an intermediate CA from a root CA in the client's certificate store and can issue a certificate on-the-fly
An attacker has received a certificate for the www.acme.org domain from a different CA, the CA will of course have to have less than perfect routines or by it's nature be open to being pressured, e.g. from a government agency

Best practice rules to thwart these kind of man in the middle attacks

Rule 1 - Use private SSL certificates for enterprise server side applications
The certificate or the private CA will then have to be installed into the certificate trust store manually.

For consumer applications this is of course impractical, but for server side applications it's quite easy e.g. installing a certificate into the java keystore or appropriate cryptolib for the platform.

Rule 2 - Verify that the complete certificate chain matches the expected
This is especially important if the Rule1 cannot be followed for some reason, e.g. client side of which limited control can be asserted. Don't rely on a connection just because you don't get a certificate exception.

Rule 3 - Start from an empty trust store for enterprise applications
My strong recommendation is that all enterprise applications should ensure that the local trust store is empty when a system is first commissioned. As services are deployed, only the certificates needed for the service is added to the trust-stores. This is perhaps only practical for server-side applications.

The inspiration for this blog post came to a large degree from this Security now podcast episode which is a great general purpose security podcast.

The wonders of datastructures...

2010-09-23T00:11:00.004+02:00

I came across this interesting post, it collates a few advanced data structures.

For normal-scale applications the stuff in your platform of choice is usually good enough, the normal lists, maps, collections of various space/time complexity implementations.

The post brings up interesting references to how moderna CPU-architectures can affect the standard space/time tradeoffs moot.

When dealing with problems to be solved at massive scale these things will likely kick-in.

Anyways, a little of the joy felt when taking the first CS-data structure classes came upon me when reading this. If I get time I'll try to experiment with a few of the implementations.

Mechanical turk and regional wage pressure

2010-09-04T00:17:00.007+02:00

Interesting post on AWS "Mechanical Turk" service, really making a strong case for extending to the service to include robust reputation mechanisms into the service to have the wages go up.

http://behind-the-enemy-lines.blogspot.com/2010/07/mechanical-turk-low-wages-and-market.html

I get the feeling though that one of the reasons that the price of labor on the mechanical turk is not only the economics discussed in the article. I believe that there is also an element of the "turks" being from other low-wage countries or regions and will accept working for less and thus pushing the prices down.

yEd review & scalability

2010-09-03T00:04:00.008+02:00

Tried out the yEd diagram editor and as an example made a picture of the scalability patterns in the previous post.

Heard some good things about it on the javaposse podcast, but I was not very impressed. It's a bit too primitive and not user friendly in it's current form. Also the included symbol library is very limited. Maybe given time it can grow on you.

Pros

It's all java so it can run anywhere, even as a webstart application so you don't need to install anything
Plenty of auto-layout styles
Quite fast

Cons

Small symbol library
Inflexible in positioning
User friendliness is not the best

Anyways, I'll continue using it for a while and see if it grows on me as I find out more of the features and get used to the user interaction model.

Scalability patterns and an interesting story...

2010-08-28T01:24:00.021+02:00

I read a pretty interesting war-story from solving scalability the pragmatic and incremental way.

http://highscalability.com/blog/2010/8/23/6-ways-to-kill-your-servers-learning-how-to-scale-the-hard-w.html

When the whole story started the blogger was a somewhat naive and inexperienced developer from the sound of it, discovering some of the fundamental scalability strategies along the way. And to be honest, we've all been there.

Fundamentally the following strategies can be used to achieve scalability in my experience, different problems lends itself to varying combinations and flavors of the scalability patterns below, but the fundamentals stay the same.

Load distribution - Spread the system load across multiple processing units
Parallelization - Work on the same task in parallel on multiple processing units
Relaxing of data constraints - Many different techniques and trade-offs with regards to the immediacy of processing / storing / access to data fall in this strategy
Queuing and batch - Achieve efficiencies of scale by processing batches of data, usually because the overhead of an operation is amortized across multiple request

The load distribution pattern really exists in two common flavors:

Load balancing / load sharing - Spreading the load across many components with out regard to the data inside of the request according to some load balancing algorithm
Partitioning - Spreading the load across many components by routing an individual request to a component that owns that data specific to the request

Partitioning can further be divided into the two patterns:

Vertical partitioning - Spreading the load across the functional boundaries of a problem space, separate functions being handled by different processing units
Horizontal partitioning - Spreading a single type of data element across many instances, according to some partitioning key, e.g. hashing the player id and doing a modulus operation, etc. Quite often referred to as sharding.

So a lot of different ways to approach scalability problems which I expect to return in future blogposts.

The lessons (with the titles sometimes paraphrased) from the blog post and some comments.

Lesson #1 (Put Smarty compile and template caches on an active-active DRBD cluster with high load and your servers will DIE!)
This is not really addressing scalability, more like enhancing performance

Lesson #2 (Don't use out-of-the-box configurations)

This is not really addressing scalability, more like enhancing performance

Lesson #3 ( single points of contention will eventually become a bottleneck)

Their solution was to introduce master-slave replication.

This is an example of the load distribution pattern by spreading the read-requests over multiple databases.

This is also an example of using the relaxing of data constraints pattern as the replication to achieve good scalability in all likelihood was not completely synchronous so the replicated data was not perfectly consistent in time

Lesson #4 ( plan in advance )

Hope for the best but prepare for the worst!

Lesson #5 ( offload your databases as much as possible )

They introduced memcachd to offload their databases from load.
A common way to use a memcachd architecture is to partition the memcached instances to hold a subset of the data elements.

This is of course yet another common example of the pattern relaxing of data constraints.

Lesson #6 ( file systems matter and can run out of space / inodes)

So all good stuff!

Addressing scalability problems in most shapes and forms need to employ one or more of the patterns outlined above.

Oracle and the OpenSolaris distribution ... RIP!

2010-08-05T00:17:00.006+02:00

First a matter of terminology, there is "Opensolaris" the open source code and community and there is "OpenSolaris" the binary distribution that Sun started to produce a couple of years back and also offered commercial support packages for.

I've been running OpenSolaris on my home machine for the last couple of years quite happily for the most part. After going through the initial hurdles to get DVD/MP3 et al to work life has been good. Now I'm tentatively starting to migrate to some linux distro given the state of things highlited below.

The last official release was made in june 2009 thus named OpenSolaris 2009.06.

Since last summer I went over to the development repo to get a sneak peak at the latest and greatest and it's been running very smoothly and including core-innovation as well as updated 3de party apps.

The next release was scheduled to be 2010.03 and the repo was allegedly only 1 build short of the final bits, but then the BIG merger happened and Oracle took control of Sun. After that no news were forthcoming and no binary distro!

Now Oracle have a totally different approach than Sun to communication and development, unarguably more commercially successful, but also extremely closed. Keeping even it's own opensource care takers in the dark, and generally the non-Oracle part of the Opensolaris community in a bit of a chaos, see for example the entries below:

Ben Rockwoods blog
News on Opensolaris Govering Board
Sporking of Opensolaris
Inside analysis

Anyways my conclusions from all this is and other sources that
a) Oracle will not sponsor an opensource Opensolaris based distribution to any meaningful degree
b) The new code contributions developed by Oracle engineering will be judged on a case-by-base whether they will go into the open or be kept behind closed doors, so far it's been put "out there", but will it continue?
c) Non-oracle interest in participation in Opensolaris community will vane and be limited to a few companies that have created products around the innovative technology in solaris

The question is will the community be strong enough to create a viable binary distribution without Oracle contribution?

I don't think so and are hedging my bets and preparing to move to some Linux distro. Any suggestions?

Hello world!

2010-07-27T01:28:00.000+02:00

For a long time now I have thought about dipping my toes into the blogosphere, being a creator an not only a consumer. During my summer-vacation the ideas have gathered and the energy reserves been charged, so time to dig in!

The content of the blog will focus around software, a few non-software related posts will likely make it's way in here as well. Should they start to eclipse the "stuff", I might move them to a separate blog.

The content will mostly be in English to be able to reach a wider audience as some of my personal contacts and colleagues are English speaking. Not a native English writer you will have to bear with me. Occasionally a post might be in Swedish if it's only of interest to such an audience.

A "project" that I have thought about exploring is to take a "grand tour" of the enterprise technology stack with opinions, observations, reviews and technology demonstrations. I have outlined a small enterprise / internet application that I will use as the skeleton for some of this. The outline of which will likely be my first "real" post.

As always in the blogosphere polite interaction and communication is greatly appreciated.

Cheers!