Client-side compression techniques

There are lots of different ways to compress data, here are a few examples of common algorithms:

Index Compression
Values are saved in a flat store with an index number referencing each value. The more values that are the same the better results you will get with this.


Shared Dictionary Compression
Similar to index compression but the dictionary file of values is stored externally from the compressed string, which greatly reduces the size, but requires the dictionary file to be downloaded and stored for compression and decompression.


Huffman Compression
Loops through every single character value and creates a probability of how often it is used, then it creates a tree of values which can be navigated by switching left or right in the tree.



LZ Compression
Using the zip algorithm which matches repeating patterns in your binary data. This gives even better better results than index compression but require more cpu and time.

Each compression technique has different advantages and disadvantages depending on your data type and processing power available. However the options are reduced when you need to run these algorithms in the browser.


I've created my own version of Index compression which supports strings, arrays and json:
http://jsfiddle.net/kmturley/MCzN7/13/



JavaScript performance tests of different methods and libraries:
http://jsperf.com/json-compression/5

From my tests the JSONH library is not only the fastest library to pack and unpack, but it created a fairly small size string and the library is not very large. So I would choose this over any other method for compression client side.

Here's a real world example of compressing data using JSONH, then storing it in LocalStorage for retrieval later on:


function set(key, value) {
    var item = JSONH.pack(value);
    localStorage.setItem(key, JSON.stringify(item));
    return item;
}

function get(key, value) {
    var item = JSON.parse(localStorage.getItem(key));
    return JSONH.unpack(item);
}


var packed = set('people', data);
var unpacked = get('people', packed);

And a real-world example showing 58% of the size after compression which takes around 0.26 of second:
http://jsfiddle.net/kmturley/8cMY5/5/

Future of Web Apps London - Day One

Today I attended Future of Web Apps London.

It was really interesting and thought I would share my notes from the first day. Looking forward to tomorrow's speakers!

Most interesting points of the day
Future of content is orbital content
Design should be focused on the content and always start with the smallest screen first.
Measure your overall performance using 'the K factor', a * b * c * d
Release, build, measure, learn, release again in quick cycles
Shared hosting will never be fast
Perception of speed is more important than the speed itself
Identify the biggest problems backed with data
Solve in limited time scale and then prove it
Create an experiment to prove a hypothesis
Test your biggest hypothesis first
Test with real users in their environment
TV across the room appears the same size as a smartphone at arms length, so actually need similar design solutions
Social is used less and less, the larger the device is
Governments are freaking out over the use of modern technology
Western senior politicians are from traditional jobs, e.g. doctors, lawyers, compared to China politicians who are mostly engineers
We all have robot brains which we rely on, our smartphones.
Smartphones are intimate devices, we use 10% of our brain thinking about our phones at any time.  
We are in a future only we understand
We are designing to get insides peoples heads
Controlling the psyche of people through their robot brain.
Different cultures need different optimisations.
Software makes enormous cultural influences e.g. Facebook imposes openness and lack of privacy.
We are ruling the world with the technology we are creating and governments are scared
Make sure you see the real world effects of technology on people with whatever you create

______________________________________________________________

Full notes from each speaker

Jeffrey Zeldman - A list apart
Content is a design problem
Everyone is designing experiences, even programmers
Design is only complete when there's nothing left to remove, especially sidebars!
31% of users are using mobile devices
Content is always the focus e.g. Instapaper, readability
Advertising and subscription money does not normally cover costs of generating content
Future of content is orbital content
Designers now aren't in control of the content e.g. browser plugins, api's, future design changes
Loose designs are better anyway for visually impaired
Ideal viewports are just a consensual hallucination
Users bypassing the design is no longer a minority, custom stylesheets, apps to scrape content
All driven by users need for content
Design is now getting the right user, the right content, at the right time.
Progressive enhancement, get the basics right first
Responsive design but always start with the smallest screen first.
Content first, design out from that, 320px up

Simon Cross - Facebook
How to create a Viral effect, look at Causality loops, chain effects
First Map, measure and optimise your traffic
Shares drive traffic, which drives growth
Identify traffic sources, social vs non social
Whats the ratio of shares vs impressions
Ratio of facebook share impressions to fb shares
Ratio of facebook clicks to site impressions
Amount of dropoff on login screens
Ratio of logged in users to not logged in
Login dialog impressions
Accepts vs declines
Measure your overall performance using 'the K factor', a * b * c * d
A = % of shares per visit
B = Shares per user per visit
C = Users from social networks
D = % of authed users
Measure this per app platform, device and publish type, like, comment etc
Monthly unique visitors, monthly auth'd users, daily auth'd users, daily published actions
Social logins, make them prominent, detect if referrer is facebook then show overlay to get them to log in
70% of users auth'd within overlay as the message was customised to their network
Auto login if available
app to user notifications
Easy lightweight social actions such as like, add to favourites, save, send
Focus on user retention

Giorgio Sardo - Microsoft
Desktop is now touch
New pointer events api
W3 standard
Multi-touch, pointer type, pen, etc

Jonathon Howell - Huddle
Return of the client app
Moving towards web apps with always on internet
Mobile has led swing back to native apps for performance
Each technology has advantages and disadvantages
There is always a trade off
Cloud apps are better than web apps
They both have centralised data and states
But user interface is different based upon device and can be native if it requires it
Build cloud services using rest apis and oauth2
Use mvc code client side to create the user experience

Sujay Tyle - Scopely
Social games have lower acquisition costs than other methods
Social integration is key
Virtual goods and in game purchases are the best way to monetise an app at the moment
Role playing games and gambling are the most profitable of those
This will increase as passbook and google wallet are integrated into iOS and android
Nordstromcom agile app video          
Release, build, measure, learn, release again in quick cycles
Make design principles applicable to success metrics.
Fail fast and often.

Drew McLellan - Edge of my seat
How quickly your site responds.
The ease the user moves from a to b
Performance is a feature
Cheap hosting is less performant
Shared hosting will never be fast
Run each section of code once and cache the results
Cache depending on how often the content changes
Convention over configuration
Varnish cache layer.
Split requests over sub domains to improve browser request performance.
Avoid cookies which add bulk to each request and bust caches.
Use cdn to remove traffic like cloudfront.              
Configure the resource headers e.g. Expires
Gzip content up to 70%. Apache mod deflate
Move Javascript to bottom of page
Perception of speed is more important than the speed itself
Use a script loader. Script.js, control.js, google loader
Use Google page speed and yahoo yslow.
Load other api scripts after your scripts
Eewei Chen - Ex-Sky
Design should also be lean and agile
Hacking agile. Lean start-ups are the future of agile
Build, measure, learn.
Become part of the future or you will die out
Innovative but applicable solutions
Identify the biggest problems backed with data
Solve in limited time scale and then prove it
Understand roles and responsibilities of the people you are working with.
Shared understanding by information shown to everyone live
Golden circles, start with why, then how, then what
Always have an elevator pitch.
Use pirate metrics. Aarrr.
Business model canvas mixed with lean start-up
Assess output, outcome, impact
We believe [output] by [assumption] by [impact]
Empathy map your assumptions. Impact vs how easy something is
Use split tests, analytics accurate track where that success came from
Fenyman, the key to science
Create an experiment to proove a hypothesis
Guess or assume, consequences, experiment
test your biggest hypothesis first
test with real users in their environment

Bruce Lawson - Opera
Media queries for monochrome
Remapping physical to virtual pixels
Css zoom viewport spec
Rem css units                        
Flex box, vertical centering with margin auto
Nested media queries
Future support for detecting js, hover, pointer, touch, interactive, keyboard, remote
TV across the room appears the same size as a smartphone at arms length, so actually need similar design solutions
Social is used less and less, the larger the device is                          
Retina images we have hope with new css hints  

Ben Hammersley - Futurist
Looking at the political and social implications of technology
Governments are freaking out over the use of modern technology
Moores law where power doubles every 12-18 months        
Exponential growth
Western senior politicians are from traditional jobs, e.g. doctors, lawyers.
Compared to China politicians who are mostly engineers
Internet technology has stormed through every other industry.
We all have robot brains which we rely on, our smartphone.
Smartphones are intimate devices, 10% of brain thinking about phone.  
We are in a future only we understand
We are designing to get insides peoples heads
Controlling the psyche of people through their robot brain.
Smart city, sensors and internet becoming very cheap
There will be City API's to create apps for people.
current examples are GPS sensors on the bus telling you its late
Pavement pedestrian density sensors in SF with a heat-map
Optimise your life using the data
Different cultures need different optimisations.
Software makes enormous cultural influences.
Facebook imposes openess and lack of privacy.
We are ruling the world with the technology we are creating.
Make sure you see the real world effects of technology on people.

Google mod pagespeed

If you are looking to increase the speed and performance of your website, Google have just release an Apache server module which improves page loading times by changing the way resources are delivered.

I haven't given it a try yet but people are reporting pages loading at almost twice the speed! That will make a big difference to your audience and definitely worth installing:
https://developers.google.com/speed/pagespeed/mod

Improving JavaScript animation

jQuery might be a great tool for quickly putting together animations on your pages. However when start to do more professional complex animations the limitations become apparent.

Previously Adobe Flash was the best solution for a complex animation, and the Flash library Greensock TweenLite was our favourite. Thankfully they have now ported the library over to JavaScript and it brings some great performance benefits.


It is 5x faster than jquery using their speed tests
http://www.greensock.com/js/speed.html

You can read the docs and download it at:
http://www.greensock.com/get-started-js/

Google Developer Make The Web Faster

While searching for articles about improving performance in websites I noticed Google have updated the 'Make The Web Faster' page with some interesting new statistics.

• The average web page takes up 320 KB on the wire.
• Only two-thirds of the compressible material on a page is actually compressed.
• In 80% of pages, 10 or more resources are loaded from a single host.
• The most popular sites could eliminate more than 8 HTTP requests per page if they combined all scripts on the same host into one and all stylesheets on the same host into one.

It's also a great resource for tips and tutorials to speed up your pages.

https://developers.google.com/speed/articles/