Testing Geo Location in iOS and Android apps using Google Maps #2

This blog has moved to https://blog.sverrirs.com/2015/11/testing-geo-location-in-ios-and-android_2.html

You will be automatically redirected.

Testing Geo Location in iOS and Android apps using Google Maps

This blog has moved to https://blog.sverrirs.com/2015/11/testing-geo-location-in-ios-and-android.html

You will be automatically redirected.

Expanding the Météo weather forecast site (#3)

I've discussed the work I've done to build a simpler weather UI for the French météo service in earlier posts (first entry and second entry).

We've been having exceptionally hot weather and I decided to expand the current web app with more information that is provided by the French weather service.

Try it

Severe weather warnings

The Météo service publishes a separate list of areas that have been issued with severe weather warnings (vigilance météorologique).  I thought it might be helpful to have this information readily available to the user.

After fruitlessly attempting to scrape the HTML on this website and being unable to locate the same elements that the browser debuggers indicated were rendered. I discovered to my dismay that most of the content on these pages are being rendered by a client side JavaScript. Why, I will probably never understand as the code is quite the mess to look at.

But luckily I noticed that the whole rendering depended on a single XHR call to an external XML service that luckily didn't require any sort of CrossSite authentication. So I simply went straight to the source and saved countless hours of debugging and fiddling with bad HTML. (http://vigilance.meteofrance.com/data/NXFR33_LFPW_.xml)

Unfortunately the source doesn't really represent the data in any easily understandable way so considerable digging around in the JavaScript code was needed to fully understand all the numerical codes presented in the XML.

The raw XML data from the Météo service

The rest of the JavaScript code and HTML gave me the pieces to understand the colour coding and images to render for each risk type and severity level.

Severe gales / strong wind

While digging around and trying to figure the weather warnings out I realized that my earlier design did not account for displaying warning and windspeed numbers for any forcasted and expected wind gales. This information was already available from my earlier parsing project and needed only minimal tweaking to get correct.

This information is now presented either within the day or as a whole day event, depending on which is applicable.

The new information elements.
Click for a larger version

Try it

This article is also available on LinkedIn:
https://www.linkedin.com/pulse/expanding-météo-weather-forecast-site-sverrir-sigmundarson

Upgrading the UI for the Météo weather forecast site (#2)

After building the initial version of the Météo weather page and confirming that the data was all correctly handled I went about sprucing the UI up a bit.

Try it

This project has been updated, click here for part three (data enhancements) of this article.

Not only was the original design purely a functional one and not particularly attractive or user friendly but it was also missing a few key data points such as intra-day weather and likelihood of rain.

The Old UI

The tools

With the help of JQuery and the JQuery UI I managed to wrestle it into a usable and aesthetically pleasing look and feel.

I ended up using the JQuery Accordion without too much customization. This widget suits the website quite well as I wanted to be able to show an overview of the next 10 days at a glance but then allow the user to drill down into each day to see the intra-day forecast in more detail.

The final UI design

The accordion is closed upon first loading the website but then the user can tap each day to drill down and see more detail data.

The initial view of the website
Accordion is closed

User has expanded a day and intra-day forecast is visible
Accordion is open

For longer term forecasts additional information is available on how confident the Météo weather model is about the prediction as well as a rough prediction of the likelihood of precipitation.

Extra data points are available for days further into the future

Customizing the forecast area

A new addition was the setup screen. This screen can be used to change the forecast area to a different city or area. This page is available by either clicking the name of the current area at the top of the page or scrolling all the way to the bottom of the page and clicking the "Configure" link.

Try it

The Setup Screen
The user can type into one of the three textboxes

Suggestions are offered as the user types.
Both place names and zip code entries are supported

Feel free to try this forecasting site out and even bookmark it for future use

Try it

This article is also available on LinkedIn
https://www.linkedin.com/pulse/ui-upgrade-météo-mobile-weather-forecast-site-sverrir-sigmundarson

Getting hassle free and fast(er) weather forecasts in France (#1)

The Météo-France Android app has been annoying me for the past 6 months with its excessive battery usage and frustrating UI navigation and experience. Finally this week I had enough and decided to come up with something simple that still gives me the same information on my mobile as the rather excellent France Meteorological office prepares and publishes.

Try it

This project has been updated, click here for part two (that discusses the UI redesign) and part three (data enhancements) of this article.

Design

As the French Météo does not publish any of its data in an easily programmable way I decided to do simple screen scraping of their existing forecast website. This is straight-forward enough to do in PHP and actually made considerably easier using the SimpleHTMLDOM library. I highly recommend it. It is the closest I've come to having BeautifulSoup in PHP.

The service is split into two pages: 

API

Scapes and re-renders the Météo data into either a JSON or JSONP format.

Try it

Supports the following URL parameters:

Parameter	Description
&area=	Lowercase name of the region or area you're interested in. E.g. strasbourg, eckbolsheim  or saverne. Defaults to "strasbourg"
&zip=	The zip code for the area. This should correspond to a zip code available in the area used. Defaults to "67000"
&callback=	Optional, if used then the call becomes a JSONP response and this value will hold the name of the client side Javascript function that should be called when the call returns.

Relies on support data from the following resources:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/img/spriteCarte40Uvs.png
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/img/spriteCarte40Temps.png
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/legend.css

Website presentation

Simple HTML/Javascript front on top of the forementioned API functionality. Supports both AREA and ZIP parameters described above.

Try it

Examples

Default call in HTML format:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/index.php

Eckbolsheim weather info in HTML format:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/index.php?area=eckbolsheim&zip=67201

Saverne API response:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/api.php?area=saverne&zip=67700

Default API response with JSONP callback:
http://labs.coruscantconsulting.co.uk/strasbourg/meteo/api.php?callback=weatherdatafunction



This acticle is also available on LinkedIn:
https://www.linkedin.com/pulse/getting-hassle-free-faster-weather-forecasts-france-sigmundarson

Base64 encoding image files using Python

In an earlier entry I mentioned that I was building a little app to display a bunch of static information. Sometimes I find it immensely useful to embed all resources into a single data-file. Not only does it make handling the data simpler but also retrieving it over the internet.

My data has a bunch of text and a few images as well to go with it. I wrote the code below (in python) to automate embedding of image data directly in with the other text that I will be serving.

@staticmethod
def encode_image_as_base64( image_path, base_path ):
    """ 
        Loads an image from the supplied path (using the base path as a 

        reference) and returns it as a base64 encoded string
    """
    
    # The image path can be a URI with a query string. 

    # Remove any query string elements, basically everything following 

    # a question (?) mark
    qs_split = image_path.split("?")
    image_path = qs_split[0]
    
    file_name = os.path.join( base_path, image_path)
    file_name = urllib.unquote(file_name)
    
    print "Encoding image: "+file_name
    
    encoded_string = ""
    with open(file_name, "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read())

    return encoded_string

It may not compress all that well but it makes data handling clean and simple.

Calculating File Change List using SHA256

Realised today that I needed a good way to create a change list document for a small app that I am writing. The app has a few hundred screens, each of which is supported by a single data document that contains the information that the screen is showing.

This data is downloaded in individual JSON files from the web the first time the user launches the application. Each of these JSON files can be amended after the app is released. I wanted to be able to provide a quick and easy way for the app to download changelist information to detect if anything needed updating.

I wanted the data to be separated in such a way that if only one file is changed the user only needs to download that one piece of data but not the rest. This is crucial as the entire dataset is just shy of 100MB whereas each individual file averages around 400K.

Python to the rescue

Wrote a little script that creates a single changelist.json file for each of my data files using the built in SHA256 key generation library that ships with python. So simple and quick:

out_file_path = r"C:\outfiles"
jdata = {}
for path, subdirs, files in os.walk(r"C:\myfiles"):
    for name in files:
        if( not name.endswith(".json") or path == out_file_path ):
            continue
            
        json_file_path = os.path.join(path, name)
        
        if( json_file_path == out_file_path ):
            continue
        
        jdata[name.replace(".json", "")] = hashlib.sha256(open(json_file_path, 'rb').read()).hexdigest()
        
out_file_name = os.path.join(out_file_path, "changelist.json")
print "Writing "+str(len(jdata))+" change list items to output file '"+str(out_file_name)+"'"

# Now write the file to the out location and quit
with open(out_file_name, 'w') as outfile:
    json.dump(jdata, outfile, indent=4, separators=(',', ': '))

Python, gotta love it :)

Layouts in Android not switching correctly between portrait and landscape

After having followed all the examples on the Android developer site to the letter about supplying multiple layouts for different screen sizes and orientation I was still having issues with them on extra large screens (10" tablets).

Basically the layout would not load when I rotated the tablet while the application was running.

If I started the application in portrait then that layout was correctly selected and when I started the app in landscape that layout was likewise correctly loaded. But after one layout was loaded the orientation change did not trigger Android to switch automatically.

The culprit was in my AndroidManifest.xml file. For some reason when I first created my activity the configChanges attribute had been set automatically for me to:

android:configChanges="keyboardHidden|orientation|screenSize"

removing screenSize solved the issue:

android:configChanges="keyboardHidden|orientation"

Obvious eh...

Edit: This behavior is actually documented in the android development guide.

Fast Flicking Between Images on Android

So I wanted to be able to have a simple control that supported swapping between a list of images as the user swiped/flicked their finger across the image.

My initial solution was to have a HorizontalScrollView which showed all my images in a row. This worked pretty well but I wasn't able to accurately stop to show each image on the screen as the user flicked between them. They always seemed to end up half off-screen.

After spending a few hours attempting to wrestle the scrolling events in the HorizontalScrollViewer under my control I realised that I must be doing something wrong, surely this isn't this difficult.

ViewPager

And yes, it couldn't have been simpler. More information on the android blog. But basically I ended up creating a simple PagerAdapter like so:

public class ImagePagerAdapter extends PagerAdapter 
{
 Bitmap[] _images = null;

 public ImagePagerAdapter( Bitmap[] images ) 
 {
  _images = images;
 }

 @Override
 public int getCount() 
 {
  return _images == null ? 0 : _images.length;
 }

 @Override
 public boolean isViewFromObject(View view, Object object) 
 {
  return view == ((ImageView)object);
 }
 
 @Override
    public Object instantiateItem(ViewGroup container, int position) 
 {
  ImageView img = null;
  
  if( position >= 0 && position < _images.length )
  {
   final Bitmap bmp = _images[position];
   
   img = new ImageView(container.getContext());
   img.setLayoutParams(new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));
   img.setImageBitmap(bmp);
   img.setScaleType(ImageView.ScaleType.CENTER_CROP);

   ((ViewPager)container).addView(img, 0);
  }
  
  return img;
    }

    @Override
    public void destroyItem(ViewGroup container, int position, Object object) 
    {
     ((ViewPager)container).removeView((ImageView)object);
    }

}

Then in my Activity/Fragment class it was a simple matter of just assigning the adapter to my pager and that is it!

ImagePagerAdapter adapter = new ImagePagerAdapter(bitmaps);
((ViewPager)v.findViewById(R.id.fragment_bird_banner_viewpager)).setAdapter(adapter);

Lesson: If things that you feel should be basic are becoming too complicated and taking too much time. You're probably doing them wrong. Step back and search/read a little bit. :)

Is my mobile app really free when it is serving ads?

I use this rather handy app, BaconReader from http://onelouder.com/ to browse Reddit.com.

I connected my tablet to my computer and turned on LogCat as I was about to start debugging my own little app. Instead two little log lines caught my eye:

D/baconreader(25507): Message Service Started
D/baconreader(25507): Checking for messages

That was something I thought shouldn't be running at all, I launched the BaconReader app to double-check that all settings related to notification and background activity for that app were indeed turned off  or set to manual (I do this in effort to conserve battery power). They were all off but still the app was periodically starting up a service to check for messages (thanks for that).

While doing this I glimpsed how much excess network activity the application was performing while I was under the impression that it was "idle" (i.e. I wasn't interacting with it).

Usage and User Profiling

The application is using what seems two different platforms to track users and usage patterns Flurry and Google Analytics. Fair enough, nothing too fishy about that I guess. Both services seemed to communicate very sparingly with the server with only an initial message sent when the application started up and then stayed silent while I let the app sit "idle".

Advertisements

I have no problems with apps that serve ads in exchange for me being able to use them for free. What I found curious was that this app seems to be contacting three different ad services with varying levels of details about both me and my device.

Millennial Media
This component, initiated a HTTP request twice every second, each time with a very verbose URL. Below is an example of the query parameters for one request:

accelerometer:true
adtype:MMBannerAdTop
ar:manual
bl:46
cachedvideo:false
conn:wifi
country:GB
density:2.0
dm:Nexus 10
dv:Android4.2.2
hdid:mmh_94888091071502DC8F18CE0A08CBFA79_6AC9AD8BDA218890306A1DAF963D603E089F82C9
height:53
hpx:2464
hsht:53
hswd:320
language:en
loc:false
mmdid:mmh_94888091071502DC8F18CE0A08CBFA79_6AC9AD8BDA218890306A1DAF963D603E089F82C9
mmisdk:4.6.0-12.07.16.a
pkid:com.onelouder.baconreader
pknm:BaconReader
plugged:true
reqtype:getad
sdkapid:62626
space:16810389504
ua:Mozilla/5.0 (Linux; U; Android 4.2.2; en-gb; Nexus 10 Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30Nexus 10
video:true
width:320
wpx:1

I'd love to know the reason why they need to know why my device is plugged in or not, how much space is free on it and what connection method I'm using to connect to the internet. Seems a little much but still nothing surprising and unexpected from ad requests. The standard tracking-cookie for my device is even included in the hdid variable (I assume) but no GPS information was transmitted (even though GPS was enabled for apps on my tablet).

Google AdMob

Pretty standard AdMob (old doubleclick.net) requests really. Nothing unexpected here, they even try to minimise the amount of characters traveling over the network. How nice of them! Still they sent two requests every second. Below is an example of their querystring:

preqs:0
session_id:6944941379333807684
u_sd:2
seq_num:1
u_w:800
msid:com.onelouder.baconreader
js:afma-sdk-a-v6.2.1
mv:8016014.com.android.vending
isu:94888091071502DC8F18CE0A08CBFA79
cipa:1
bas_off:0
format:320x50_mb
oar:0
net:wi
app_name:38.android.com.onelouder.baconreader
hl:en
gnt:0
u_h:1232
carrier:
bas_on:0
ptime:0
u_audio:1
u_so:p
output:html
region:mobile_app
u_tz:60
client_sdk:1
ex:1
slotname:a14eb80d44ba957
caps:inlineVideo_interactiveVideo_mraid1_th_autoplay_mediation_sdkAdmobApiForAds_di
jsv:46

Ad Marvel

This by far was the worst offender. AdMarvel was a small mobile ad startup that was recently (in 2010) bought by Opera Software. The device performed 4 requests every second (where two requests were identical except for one querystring variable had changed, that is retrynum had been incremented from 0 to 1). Below is an example of one of their query string:

site_id:23206
partner_id:7b862efbe6c75952
timeout:5000
version:1.5
language:java
format:android
sdk_version:2.3.7.1
sdk_version_date:2013-02-04
sdk_supported:_admob_millennial_amazon
device_model:Nexus 10
device_name:JDQ39
device_systemversion:4.2.2
retrynum:0
excluded_banners:
device_orientation:portrait
device_connectivity:wifi
resolution_width:1600
max_image_width:1600
resolution_height:2464
max_image_height:2464
device_density:2.0
device_os:Android
adtype:banner
device_details:brand:google,model:Nexus 10,width:1600,height:2464,os:4.2.2,ua:Mozilla/5.0 (Linux; U; Android 4.2.2; en-gb; Nexus 10 Build/JDQ39) AppleWebKit/525.10+ (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2
hardware_accelerated:true
target_params:UNIQUE_ID=>f780070d9537897a||GEOLOCATION=>59.4954954894955%2C-0.99682319125740403||bucket=>4||subreditname=>Front+Page||RTBID=>FBATTRID%3Ae939e527-35db-4e98-8d73-e05fead999b1||APP_VERSION=>2.8.1||RESPONSE_TYPE=>xml_with_xhtml||BNG=>0

What I find worrying here is (besides the size of the request) their final target_params variable which not only includes the ID for my device and exactly what part of the application I am viewing but also a pretty accurate GPS location (disabling GPS access for all apps got rid of that) and what looks to be Facebook related FBATTRID tag to serve ads from Facebook's MoPub Marketplace.

Talk about being tracked between devices.

Conclusion

All of this activity was happening multiple times a second for all three advertising services on my device. So six times a second my device was communicating tracking information and requesting advertisements back.

One of the most battery draining activities is transmitting data through the air on a mobile device. Based on these three services it seems like the industry standard is to query twice every second which to me feels overly aggressive and costly on my battery charge.

So what are we really getting for free? How much money are we spending on electricity to power our devices that is then used directly to serve us advertisements?

Perhaps I should just buy the thing, probably cheaper in the long run!

HTC Desire S USB Driver on Windows 7 (64bit)

This blog has moved to https://blog.sverrirs.com/2013/04/htc-desire-s-usb-driver-on-windows-7.html

You will be automatically redirected.