Reverse Engineering Evernote Penultimate (or: When is a picture not a picture?)

In this post Alex Caithness takes a look at “Penultimate” on the iPad and discovers that a picture paints a thousand words… but only once you work out how that picture is stored.

Recently we came up against an iPad note-taking application by the name of “Penultimate”. As a user it’s actually really quite a nice app to use: you can create multiple “notebooks” each with as many pages as you require on which you can jot down notes, draw diagrams or just doodle with a stylus or your finger. For a user it’s nice, simple and intuitive, for an intrepid forensic analyst, it’s not quite as straight-forward…

Penultimate Hello World Screenshot

Penultimate Screenshot

After installing the application on a test iPad, creating a couple of notebooks and having a doodle in each of them, I performed an extraction of the device. Poking through the application’s folder, this is the layout of the files which we can acquire with a backup-style acquisition:

com.cocoabox.penultimate
└───Library
 ├───Preferences
 │     com.cocoabox.penultimate.plist
 │ 
 └───Private Documents
 │     notebookList
 │ 
 └───notebooks
 │     notebookListBackup
 │ 
 ├───23FDA4FF-CE5E-4353-8D9A-B0C3E3E8AAE7
 │     100F53F2-BD1F-4046-8104-714E42264DAE
 │ 
 └───DD47D226-0CB3-460F-A0F9-3A7E2A795B3D
       BBDC9CD6-2685-4AFF-BCB4-D370F975A3F1
       BEB76173-1F62-472B-B0DC-F69EF039A59F
       C1766626-9BD2-44F9-A2DD-D7C4F23367A8

Now, ideally what we’d liked to have seen at this point are some image files, a nice friendly JPEG, GIF or PNG or two would have been just lovely. But no, it’s not going to be that easy (and, in fairness, if it had been I wouldn’t be bothering with this blog would I?).

So what do we actually have? Well, we can see the familiar “Library” folder in which we find the (also familiar) “Preferences” folder which contains a property list, the contents of which are mostly mundane (though we do have application launch counts and dates which may be of interest in some situations). More interesting is the “Private Documents” folder and the files and folders which are contained within it.

In the root of the Private Documents folder is a binary property list file named “notebookList”. It is a ‘NSKeyedArchiver’ plist, which means, as we discussed in a previous blog, it needs some additional processing in order for us to see the actual structure of the data. Luckily I had PIP and the “ccl_bplist” Python module to hand, both of which can perform that transformation. With the data structure untangled I proceeded with digging around to see what, if anything, we could extract from the “notebookList”.

Among some less interesting data, the root object contains a list of the notebook objects which are stored by the device, under the “notebooks” key.

notebookList in PIP

notebookList in PIP

Each of the notebook objects (which are represented by dictionaries in the property list) contain the keys: ‘name’, ‘title’, ‘pageNames’, ‘created’, ‘modified’,  ‘changeCount’, ‘blankPages’, ‘creatingDeviceId’, ‘editingPageName’, ‘pageDataIsCooked’, ‘versionMajor’, ‘versionMinor’, ‘pagePaperStyles’, ‘paperStyle’, ‘imported’ , ‘coverColor’, ‘originalName’.

The first few there are of immediate interest: the value of “title” does indeed contain the title of the notebooks that I created during my testing and the “created” and “modified” timestamps also ring true. The values of the “name” keys are familiar as well, albeit for a different reason; in my test data we see the “name” values “DD47D226-0CB3-460F-A0F9-3A7E2A795B3D” and “23FDA4FF-CE5E-4353-8D9A-B0C3E3E8AAE7” – matching the directory names that were extracted underneath the “notebooks” folder (see above). Beyond this, the “pageName” key contains a list of values which also match the names of files in each of the “name” directories.

So, with the “notebookList” file we have some useful metadata and a helpful guide to how the other files are organised, but there’s still no sign of the content of the notes themselves. Delving deeper into the folder structure, our next stop is one of the files which was named in the “pageName” list mentioned above.

Opening one of the “page” files we find another “NSKeyedArchiver” property list. After unravelling the structure of the file we find a top-level object containing further metadata (including a “blankDate” which appears to match the “created” timestamp reported in the “notebookList” and the dimensions of the note) along with a list of “layers”. Each of the “layer” objects (again represented by dictionaries) have keys for the layer’s colour (more on that later) the layer’s dimensions and a list of “layerRects” – sections of the layer where the user has drawn their notes; and that’s where we finally find the image itself.

Sort of.

Structure of the page object

Structure of the page object

Each of the “layerRects” objects are represented by (and this shouldn’t be a surprise by now) a dictionary. There are two keys: firstly “rect” which contains a string of two sets of “x,y” co-ordinates: the bounds of this layerRect on the page. The second key is “values” which requires a little extra explanation. As noted in my previous post on NSKeyedArchiver files, their function is to represent a serialisation of objects; the object under the “values” key is a “CBMutableFloatArray” which is programmer talk for a list of floating-point numbers. With this in mind we quite reasonably expect to see just that – a list of floating point numbers; but no, instead we get a data field containing binary data! As nothing else in this data had been straight-forward this was disappointing, but by this point didn’t surprise me. Floating-point numbers (like any numerical data) can be represented quite happily in binary, so I set about trying to turn the binary data into a list of floating-point numbers. I extracted one of these data fields and with a nice Python one-liner did just that:

>>> struct.unpack("<18f", data)
(282.0, 589.0, 5.506037712097168, 281.8666687011719, 588.2999877929688, 5.506037712097168, 281.73333740234375, 587.5999755859375, 5.506037712097168, 281.5999755859375, 586.9000244140625, 5.506037712097168, 281.4666442871094, 586.2000122070312, 5.506037712097168, 281.33331298828125, 585.5, 5.506037712097168)

So now we can see the numbers, but what did they mean? The meaning becomes clearer if we group them into threes:

282.0, 589.0, 5.506037712097168
281.8666687011719, 588.2999877929688, 5.506037712097168
281.73333740234375, 587.5999755859375, 5.506037712097168
281.5999755859375, 586.9000244140625, 5.506037712097168
281.4666442871094, 586.2000122070312, 5.506037712097168
281.33331298828125, 585.5, 5.506037712097168

What we have here is sets of x-y co-ordinates and another value (the meaning of which will become clearer later). So, here, finally is our drawing; stored as a list of co-ordinates rather than any conventional graphics format – the combination of the points in the “layerRects” in each of the layer objects gives you the full picture.

But how to present this data? A list of co-ordinates like this is of little to no use; we want to see the image itself. So I got to thinking: how does the application treat these co-ordinates? Well, when the user draws in the application they are essentially ‘painting’: moving a circular ‘brush’ around the screen making up the lines in the drawing, so if I was able to plot a circle at each of these co-ordinates perhaps I would see the picture? This thought process also gave me an idea as to what the third value in each of the co-ordinate groupings might represent: part of what makes the drawings look natural is that the “line” that is drawn is not of a uniform width, it grows and shrinks with the speed at which the finger or stylus moves, giving an impression of weight – perhaps the final value related to this?

So how to plot these co-ordinates? First of all I looked at various Python imaging libraries (PIL and PyGame both came on my radar), but there were issues with both (especially with PIL which still lacks a proper Py3k port) so I turned my attention to an alternative solution: SVG. SVG (Scalable Vector Graphics) is a graphics format specified by the W3C; it uses XML to describe 2D graphics. Because SVG just uses XML, no imaging libraries would be required; I could simply generate textual mark-up describing where each of my circular points should be plotted on the screen. Taking the data above extracted above as an example, the mark-up would be along the lines of:

<!DOCTYPE svg>
<svg height="865" version="1.1" viewBox="0 0 718 865" 
     width="718" xmlns="http://www.w3.org/2000/svg">
    <circle cx="282.0" cy="589.0" 
            fill="#000000" r="5.506037712097168"/>
    <circle cx="281.8666687011719" cy="588.2999877929688" 
            fill="#000000" r="5.506037712097168"/>
    <circle cx="281.73333740234375" cy="587.5999755859375" 
            fill="#000000" r="5.506037712097168"/>
    <circle cx="281.5999755859375" cy="281.5999755859375" 
            fill="#000000" r="5.506037712097168"/>
    <circle cx="281.4666442871094" cy="586.2000122070312" 
            fill="#000000" r="5.506037712097168"/>
    <circle cx="281.33331298828125" cy="585.5" 
            fill="#000000" r="5.506037712097168"/>
</svg>

Each “circle” tag has an x and y co-ordinate (“cx” and “cy”) a fill colour (here expressed as an html colour code – black in this case) and a radius (“r”). Running some tests gave some good output; but things weren’t quite right, firstly, and most importantly, the image was mirrored in the y-axis, presumably caused by a difference of opinion between Penultimate and SVG as to where the y-axis originates, an easy fix (just subtract the ‘y’ value from the height of the image and use that value instead). Also, the lines in the writing all looked very chubby, making writing very tricky to read. Theorising that this was caused by Penultimate storing diameters and SVG requiring radii, I halved the value which improved things, but comparing the output with what I could see on the screen things still weren’t quite looking right so on a whim I halved the value again which made things look right (I’m not entirely sure why it should be the case that you have to quarter the value – it may be to do with penultimate adding a ‘feathered’ fade to the brush which increases it’s diameter though).

I created a proof-of-concept Python script to check that my thinking was correct and I was pleased to see that the output now matched what I could see on the iPad’s screen, save for the fact that the iPad was in glorious Technicolor and my script’s output was monochrome. I mentioned previously that the “layer” objects contained colour information under the unsurprisingly named “color” key.

The object stored in the “color” key has values for red, blue and green – each one a value between 0 and 1. Those readers who have had even the briefest dalliance with graphic manipulation programs will be familiar with colour-sliders: combining red, green and blue in different proportions in order to generate different colours, which is what these values represented. My SVG generated output was working with HTML colour codes which are made up of 3 bytes, each representing an amount of red, green and blue, this time using values from 0x00 to 0x0FF, to get colour into my output, all I had to do is multiply 0xFF by the correct value from the “layer” object’s colour fields and recombine those values into an HTML colour code. I modified my script and now the output reflected both the form and colour displayed by the App on the iPad.

Comparison between iPad and script output 1

Comparison between iPad and script output (1)

Comparison between iPad and script output 2

Comparison between iPad and script output (2)

Working on extracting data from Penultimate was a particularly enjoyable experience as it required a combination of a number of different concepts and led to the use of a number of different technologies to create a solution to automate the extraction of the data, which when it all boiled down was satisfyingly simple.

And as to the question: when is a picture not a picture? Well, quite simply: when it’s a series of serialised floating-point co-ordinate triplets representing points on a page, broken up into rectangles on a layer on a page which is stored in a NSKeyedArchiver property list file (obviously!).

If you have any questions, comments or queries regarding this post, as always you can contact us by dropping an email to research@ccl-forensics.com or by leaving a comment below.

Update: As requested I’ve uploaded the script for research purposes. You can find it here: http://pastebin.com/VYenpXUi 

Alex Caithness, CCL Forensics

New Epilog Signature files released

Epilog Signature files allow users to add specific support for new databases they encounter and although they are designed so that Epilog’s users can create their own signatures when the need arises, CCL-Forensics are committed to updating and releasing a sets of signatures, pre-written and ready to use.

In this new release we have had a real focus on smartphones adding support for:
• iOS6
• Android 4.0 (Ice Cream Sandwich)
• Android 4.1 (Jelly Bean)
• Android 3rd Party Applications
• iOS 3rd Party Applications
• Skype

We always welcome suggestions for signatures that you’d like to see added to the signature collection so please get in touch on epilog@ccl-forensics.com

For more information on epilog please visit our website – www.cclgroupltd.com/Buy-Software/

Parsing Apple System Log (ASL) files on iOS and OSX for Fun and Evidence (and a Python script to do it for you)

(If you’re dying to get stuck in and are only after the links to the Python scripts, they can be found at the bottom of the post!)

After every update to iOS I like to take a file system dump of one of our test iDevices and have a poke around to see what’s changed and what’s new. Recently, on one of my excursions around the iOS file system, I came across something that looked promising that I hadn’t dug into before: a bunch of files with the “.asl” extension which were located on the data partition in “log/DiagnosticMessages”. There were lots of them too –each with a file name referring to a particular date – they went back months!

DiagnosticMessages file listing

Log Files!

“Loads of lovely log files!” I thought to myself as I excitedly dropped one of the files into my current text editor of choice (Notepad++ if you’re interested) only to be disappointed by what was clearly a binary file format.

ASLDB File in a text editor

Curses!

So I headed over to Google and entered some hopeful sounding search queries and came across a very useful blog post (http://crucialsecurityblog.harris.com/2011/06/22/the-apple-system-log-%E2%80%93-part-1/) which described the role of ASL files on OSX and listed some ways for accessing the logs from within OSX, but I was interested in gaining a better understanding of the file format (besides, the nearest Mac to me was on a different floor!).

A little more digging revealed that the code that governed the ASL logging, and the files it generated were part of the Open Source section of OSX, as a result I was able to view the code that was actually responsible for creating the file – my luck was looking up!

The two files I was particularly interested in were “asl.h” (most recent version at time of posting: http://opensource.apple.com/source/Libc/Libc-763.13/include/asl.h) and “asl_file.h” (most recent version at time of posting: http://opensource.apple.com/source/Libc/Libc-763.13/gen/asl_file.h). C header files are great; basically, their purpose is to define the data structures that are subsequently used in the functional code, so when it comes to understanding file formats, quite often they’ll tell you all you need to know without having to try and follow the flow of the actual program. Better yet, these files were pretty well commented. I know that not everyone reading this is going to want to read through a bunch of C code, so I’ll summarise the file format below (all numeric data is big endian):

First, the file Header:

Offset Length Data Type Description
0 12 String “ASL DB” followed by 6 bytes of 0x00
12 4 32bit Integer File version (current version is: 2)
16 8 64bit Integer File offset for the  first record in the file
24 8 64bit Integer Unix seconds timestamp, appears to be a file creation time
32 4 32bit Integer String cache size (not 100% sure what this refers to, may be maximum size for string entries in the records)
36 8 64bit Integer File offset for the last record in the file
44 36 Padding Should all be 0x00 bytes

So nothing too ominous there, although all of those pad-bytes at the end of the header suggest redundancy in the file spec in case apple ever fancy changing something. Indeed the fact that the header tells us that we’re on version 2 of the file format suggests that this has already happened.

The records in the file are arranged in a “doubly linked list”, that is, that every record in the file contains a reference (ie. the file offset of) the next and previous records.  From a high level, the records themselves are made up of a fixed length data section, followed by a variable length section which allows the storage of additional data in a key-value type structure, finally followed by the offset of the previous record. The table below explains the structure in detail.

NB: The string storage mechanism the records use is a little bit…interesting – I’ll explain in detail later in this post, but for now if you see a reference to an “ASL String”, I mean one of these “interesting” strings!

Offset Length Data Type Description
0 2 Padding 0x00 0x00
2 4 32bit Integer Length of this record (excluding this and the previous field)
6 8 64bit Integer File offset for next record
14 8 64bit Integer Numeric ID for this record
22 8 64bit Integer Record timestamp (as a Unix seconds timestamp)
30 4 32bit Integer Additional nanoseconds for timestamp
34 2 16bit Integer Level (see below)
36 2 16bit Integer Flags
38 4 32bit Integer Process ID that sent the log message
42 4 32bit Integer UID that sent the log message
46 4 32bit Integer GID that sent the log message
50 4 32bit Integer User read access
54 4 32bit Integer Group read access
58 4 32bit Integer Reference PID (for processes under the control of launchd)
62 4 32bit Integer Key-Value count: The total number of keys and values in the key-value storage of the record
66 8 ASL String Host that the sender belongs to (usually the name of the device)
74 8 ASL String Name of the sender (process) which send the log message
82 8 ASL String The sender’s facility
90 8 ASL String Log Message
98 8 ASL String The name of the reference process (for processes under control of launchd)
106 8 ASL String The session of the sender (set by launchd)
114 8 * Key-Value count ASL String[Key-Value count] The key-value storage: A key followed by a value, followed by a key followed by a value… and so on. All keys and values are strings

The level field mentioned above will have a numerical value which refers to the levels shown below:

Level Meaning
0 Emergency
1 Alert
2 Critical
3 Error
4 Warning
5 Notice
6 Info
7 Debug

As mentioned, the “ASL String” data type is a little odd. The ASL fields above take up 8 bytes, if the most significant bit in the 8 bytes is set (ie is 1), the rest of the most significant byte gives the length of the string, which occupies the remaining 7 bytes (unused bytes are set to 0x00). Conversely, if the top bit in the ASL String data type is not set (ie. Is 0) the entire 8 bytes should be interpreted as a 64bit Integer which gives the file offset where the string can be found. The string will be stored thusly:

Offset Length Data Type Meaning
0 2 Padding Padding bytes 0x00 0x01
2 4 32bit Integer String length
6 String length UTF8 String (nul-terminated) The string data

In order to get a better grip of what can be held in these files I decided to create a Python module to read these files and used it to dump out the contents of the ASL files I found on the iPhone.

Running the script

Running the script

Output from the script (iOS)

A snippet of the output produced by processing an iPhone’s ‘DiagnosticMessages’ folder

The first thing that struck me after running the script was the volume of messages: 16161 log messages spanning 10 months – and this was on a test handset which had lay idle for weeks at a time. The second thing was the prevalence of messages sent by the “powerd” daemon, over 87% of the messages had been sent by this process. The vast majority of these messages related to the device waking and sleeping – not through user interaction, but while the device was idle. Most of these “Wake” events occurred 2-5 minute apart, presumably to allow brief data connectivity to receive updates and push messages from apps.

Output from the script (iOS powerd messages)

Some powerd Wake and Sleep messages

The key thing that interested me about these messages was that they also noted the current battery-charge percentage in their text: this is the sort of data that just begs to be graphed, so I knocked up a little script which utilised the parsing module I had just written to extract just this data and present it in a graph-friendly manner.

Graph Friendly powerd Data

Graph Friendly Data

After graphing it (you want to use a scatter graph in Excel for this, not line as I discovered after some shouting at my screen) you are left with a graph which gives you some insight into the device’s use.

iOS Battery Use Graph

Some iPhone Power Usage (click for full-size)

The graph above shows around 3 weeks of battery usage data from the test handset. As noted previously, this test device would lay idle for days at a time (as suggested by the gentle downward gradients) but there were periods when the handset was in use, as shown by the steeper downward gradients on the 26th and 27th of April, which mostly took place within office hours. You can also clear see the points where the device was plugged in to be charged, suggested by the very steep upward gradients. As noted the power messages occur around ever 2-5 minutes, so the resolution is actually fairly good. The exception to this is while the device is plugged in as it no longer needs to sleep to preserve battery charge; typically I only saw an event when charging began and another when the device was unplugged and the battery began to discharge again.

There are a few other messages in the iOS ASL log that look interesting, but at this time I don’t have enough nice control data to make much of them. One thing that did hearten me somewhat was the fact that on the few extractions I’ve had the opportunity to take a look at from later revisions of iOS 5, there did seem to be some extra processes that were logging messages, so it’s my hope that we’ll see more and more useful data make its way into the ASL logs on iOS.

In addition to looking at iOS ASL files, I thought I’d take a look at some from an OSX installation. Pulling the logs from the “var/log/asl” on Lion (10.7.3) and running the parsing script across the whole directory brought back a far more varied selection of messages.

Output from the script (OSX)

Variety is the spice of life.

The number of records returned was actually far less than on iOS, partially due to the iOS “powerd” being so chatty, but more crucially because OSX tidies up its logs on a weekly basis. That’s not to say that you will only recover a week’s worth of logs though – on this test machine I recovered logs spanning 7 months. Rather, OSX has short-term log files (those with file names which begin with a timestamp) which have a shelf-life of a week and long term log files (those with file names which begin with “bb” followed by a timestamp). The “bb” in the long term log’s file name presumably stands for “best before” and the date, which is always in the future, is the date that the file should be cleared out. The short term log files tend to hold more “intimate” entries, often debug messages sent from 3rd party applications; the long term logs err more on the side of system messages. One particularly useful set of messages in the long term log are records pertaining to booting, shutting down, logins and logouts (hibernating, waking and failed logins are recorded too, but they end up in the short-term logs).

(As an aside:  one of my favourite things that I discovered when looking through these logs was the action of waking a laptop running OSX by wiggling a finger on the trackpad is recorded in the logs as a “HID Tickle”. Lovely.)

Like I did with the iOS power profiling, I put together a script which extracted these login and power records and timelines them.

OSX Login and power timeline

Login and power timeline

A couple of things worth noting beyond the basic boot/shutdown/login records: firstly when the device wakes it records why it happened – this can be quite specific: a USB device, the lid of a laptop being opened, the power button being pressed, etc. Secondly, you can see terminal windows (tty) being opened and closed as opening a terminal window involves logging in to a terminal session (OSX does this transparently, but it’s still logged).

We’ve released the scripts mentioned in this post to the community and they can be downloaded from https://code.google.com/p/ccl-asl/. The “ccl_asl” script is both a command line utility for dumping the contents of ASL files as well as a fully featured class module which you can use to write scripts along the lines of the battery profiler and login timeline scripts.

ASL files are, on the one hand, fairly dry system logs, but on the other, with a little work you can harvest some really insightful behavioural intelligence. As always if you have any questions, comments or suggestions you can contact us on research@ccl-forensics.com or leave a comment below.

Alex Caithness

Geek post: NSKeyedArchiver files – what are they, and how can I use them?

If you have spent any time investigating iOS or OSX devices you will probably have come across files (usually property lists) making reference to NSKeyedArchiver. Certainly, in my experience working with iOS, these files can often contain really interesting data (chat data for example) but the data can appear at first glance unstructured and difficult to work with.

In this blog post I aim to explain where these files come from, how they are structured and how you can get the most out of them.

Remember, remember…

NSKeyedArchiver is a class in the Mac programming API which is designed to serialise data. Data serialisation is the process through which data belonging to a program currently held in memory is written to a file on disk, so that it may be reloaded into memory at some point in the future.

This process can be achieved in a number of ways depending on the requirements of the programmer; however, NSKeyedArchiver provides a convenient way for entire “objects” in memory to be serialised, so it is a widely-used method.

Let’s take a moment to consider what is meant by an “object” in terms of programming (don’t worry; I’m not going to get too programmery). Many modern programming languages allow for (or are entirely based upon) the Object Oriented Programming paradigm. Put very generally this means that they are based around data structures (objects) which contain both data fields and built-in functionality (usually known as “methods”).

So let’s imagine that we were writing the code to define a “Person” object: we might define data fields such as: “Name”; “Age”; “Height”; and “Weight” – but we might also want to give it functionality. For example: “Speak” and “Wave”.

Obviously, a “Car” object would be different from a “Person” – it would have data fields like: “Make”; “Model”; and “Fuel-Type”. It might also have a data field for “Driver” which would hold a reference to a “Person” object.

A “Road” object might have a data fields for: “Name”; “Length”; and “Speed-Limit” along with a data field containing a collection of “Car” objects (each having a reference to a “Person” object in their “Driver” data field).

Similar (and often far more complicated) data structures might be represented in a chat application: a “Message-List” object containing a collection of “Message” objects containing fields for “Sent-Time”, “Message-Text” and “Sender”, which itself contains a reference to a “Contact” object which contains fields for “Nickname”, “Email-Address” and so on.

It’s these kinds of data structures that NSKeyedArchiver converts from objects in memory and stores as a file which can subsequently be loaded back into the memory to rebuild the structure.

So what must NSKeyedArchiver store in order to achieve this? Well, there are two requirements: it has to store details of the type of object it’s serialising; and it has to store the data held in the objects (and the correct structure of the data).

NSKeyedArchiver property lists

NSKeyedArchiver serialises the object data into a binary property list file, the basic layout of which is always the same:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>$archiver</key>
        <string>NSKeyedArchiver</string>
        <key>$objects</key>
        <array>
            <null/>
            <string>Alex</string>
            <dict>
                <key>Name</key>
                <dict>
                    <key>CF$UID</key>
                    <integer>1</integer>
                </dict>
            </dict>
        </array>
        <key>$top</key>
        <dict>
        <key>root</key>
        <dict>
            <key>CF$UID</key>
            <integer>2</integer>
        </dict>
        </dict>
        <key>$version</key>
        <integer>100000</integer>
    </dict>
</plist>

Listing 1: Overview of an NSKeyedArchiver XML property list file.

In Listing 1 we have an example of an NSKeyedArchiver property list (converted to XML for ease of reading).  At the top level, every NSKeyedArchiver file is made up of a dictionary with four keys.

Two of the keys simply provide metadata about the file: the “$archiver” key should always be followed by a string giving the name of the archiver used to create this file, which should obviously always be “NSKeyedArchiver” and the “$version” key should be followed by an integer giving the version of the archiver (100000 appears to be the only valid value).

The other two keys (“$objects” and “$top”) contain the data that has been serialised and its structure.

The “$objects” key is followed by an array containing all the pieces of data involved in the serialisation, but stored flat with little or no structure. Each of these pieces of data can be understood as being enumerated starting from zero. Within this array of objects may be data which contains the structure shown in Listing 2:

<dict>
        <key>CF$UID</key>
        <integer>0</integer>
</dict>

Listing 2: an example of the CF$UID data type.

The CF$UID data type in Listing 2 is a dictionary with a single key (“CF$UID”) which is followed by an integer number (this layout is what you will see when the property list is represented in XML; in the raw binary format the “UID” data type is a separate entity which doesn’t require the dictionary structure).

These data types represent a reference to another entity in the “$objects” array. The number of the CF$UID gives the position of the array. Consider the snippet shown in Listing 3:

    <key>$objects</key>
    <array>
        <null/>
        <string>Alex</string>
        <dict>
            <key>Name</key>
            <dict>
                <key>CF$UID</key>
                <integer>1</integer>
            </dict>
        </dict>
    </array>

Listing 3: an example of a “$objects” array.

Listing 3 shows an “$objects” array containing three pieces of data. Indexing them starting from 0 we have:

  1. A null
  2. A string containing the value “Alex”
  3. A dictionary

The dictionary at index 2 contains a single key: “Name”. The following value is a “CF$UID” data type referencing index 1 in the “$objects” array so we could consider the data to be equivalent to Listing 4:

    <key>$objects</key>
    <array>
        <null/>
        <string>Alex</string>
        <dict>
            <key>Name</key>
            <string>Alex</string>
        </dict>
    </array>

Listing 4: the “$objects” array from Listing 3 “unpacked”.

This example is very simplistic; in reality the structure revealed by unpacking the object array can be extremely deeply nested with objects containing references to objects containing references to objects…and so on.

The observant among you may be thinking “this seems like a very inefficient way to represent the data”, and for this example you’d certainly be right! However, in most real-life cases the complex data held in these files contains many repeating values which, when arranged this way, only have to be stored once but can be referenced in the “$objects” array multiple times.

The “$top” key is our entry point to the data, so it is the data held at this key that represents the total structure of the object that has been serialised. This key will be followed by a single dictionary which again contains a single key “root”. The “root” key will be followed by a single CF$UID data type which will be a reference the top level object in the “$objects” array.

Returning to the example in Listing 1 the “root” is referencing the object at index 2 in the objects array. So expanding this, our complete data structure is shown in Listing 5:

    <key>$top</key>
    <dict>
        <key>root</key>
        <dict>
            <key>Name</key>
            <string>Alex</string>
        </dict>
    </dict>

Listing 5: Expanded “$top” object, showing complete data structure.

A sense of identity

So far we have only seen examples of basic data stored in this structure where the type of data is implicit but in most files you are likely to encounter you will see additional data relating to the type of the objects being stored.

Listing 6 shows an unpacked “$top” object from a “CHATS2.plist” file produced by the iOS application “PingChat”:

    <key>$top</key>
    <dict>
    <key>root</key>
    <dict>
        <key>$class</key>
        <dict>
            <key>$classes</key>
            <array>
                <string>NSMutableDictionary</string>
                <string>NSDictionary</string>
                <string>NSObject</string>
            </array>
            <key>$classname</key>
            <string>NSMutableDictionary</string>
        </dict>
        <key>NS.keys</key>
        <array>
            <string>pingchat</string>
        </array>
        <key>NS.objects</key>
        <array>
            <dict>
                <key>$class</key>
                <dict>
                    <key>$classes</key>
                    <array>
                        <string>NSMutableArray</string>
                        <string>NSArray</string>
                        <string>NSObject</string>
                    </array>
                    <key>$classname</key>
                    <string>NSMutableArray</string>
                </dict>
                <key>NS.objects</key>
                <array>
                    <dict>
                        <key>$class</key>
                        <dict>
                            <key>$classes</key>
                            <array>
                                <string>BubbleItem</string>
                                <string>NSObject</string>
                            </array>
                            <key>$classname</key>
                            <string>BubbleItem</string>
                        </dict>
                        <key>state</key>
                        <integer>1</integer>
                        <key>image</key>
                        <string>$null</string>
                        <key>msg</key>
                        <string>Yo</string>
                        <key>author</key>
                        <string>testingtesting</string>
                        <key>time</key>
                        <dict>
                            <key>$class</key>
                            <dict>
                                <key>$classes</key>
                                <array>
                                    <string>NSDate</string>
                                    <string>NSObject</string>
                                </array>
                                <key>$classname</key>
                                <string>NSDate</string>
                            </dict>
                            <key>NS.time</key>
                            <real>307828812.649871</real>
                        </dict>
                    </dict>
                </array>
            </dict>
        </array>
    </dict>
    </dict>

Listing 6: Expanded “$top” object taken from a PingChat “CHATS2.plist” file.

In Listing 6 we can begin to see how complex the serialised data can become (and this is a simpler example). However, if you keep your cool and realise that the data is still well-structured it is possible to parse the data into something more meaningful.

One new data structure we encounter for the first time here is the “$class” structure. “$class” isn’t part of the data itself, but rather information about which type of object has been serialised. This information is obviously important when the program that serialised the data comes to deserialise it, but we can also use it to give us clues about the meaning of the data; consider the snippet in Listing 7:

<dict>
    <key>$class</key>
    <dict>
        <key>$classes</key>
        <array>
            <string>NSMutableArray</string>
            <string>NSArray</string>
            <string>NSObject</string>
        </array>
        <key>$classname</key>
        <string>NSMutableArray</string>
    </dict>
    <key>NS.objects</key>
    <array>
        <dict>
            <key>$class</key>
            <dict>
                <key>$classes</key>
                <array>
                    <string>BubbleItem</string>
                    <string>NSObject</string>
                </array>
                <key>$classname</key>
                <string>BubbleItem</string>
            </dict>
            <key>state</key>
            <integer>1</integer>
            <key>image</key>
            <string>$null</string>
            <key>msg</key>
            <string>Yo</string>
            <key>author</key>
            <string>testingtesting</string>
            <key>time</key>
            <dict>
                <key>$class</key>
                <dict>
                    <key>$classes</key>
                    <array>
                        <string>NSDate</string>
                        <string>NSObject</string>
                    </array>
                    <key>$classname</key>
                    <string>NSDate</string>
                </dict>
                <key>NS.time</key>
                <real>307828812.649871</real>
            </dict>
        </dict>
    </array>
</dict>

Listing 7: Snippet of a single object in a PingChat “CHATS2.plist” file.

Let’s take a look at the objects involved here and what the “$class” sections can tell us about the data held. The “$class” structure takes the form of a dictionary containing two keys. The “$classname” section is fairly straightforward; it simply gives us the name of the type of object we’re dealing with.

So in the case of the first “$class” structure encountered, we find that the object is of type “NSMutableArray”, which a quick Google search tells us is a “Modifiable array of objects” – so the data held in this object is going to take the form of an array or list.

The other key in the “$class” structure is “$classes”; this is a little more subtle and requires a little more explanation of one of the key concepts in most object-oriented programming languages: inheritance.

Think back to the explanation of the “Person” object. A “Person” object had the fields: “Name”; “Age”; “Height”; and “Weight” and the functionality to “Speak” and “Wave”. Now imagine that we wanted to create a new type of object: “DigitalForensicAnalyst”. We would want this new type of object to have some specialised functionality: “Image”; “Analyse”, and so on.

However, a “DigitalForensicAnalyst” is a “Person” too – they have a name, they can speak and wave. Now, programmers are stereotyped as being lazy (because they are) so it is unlikely that after spending all that time writing and debugging the code to represent a “Person” that they are going to duplicate all that hard work when it comes to creating a “DigitalForensicAnalyst”.

Instead they would have the “DigitalForensicAnalyst” object inherit the functionality from “Person”; this means that only the new functionality of “Image” and “Analyse” need be created from scratch, all of the other functionality comes free thanks to this inheritance.

Coming back to the “$classes” key, this will be followed by an array containing the names of all of the types that this object inherits from. So in the case of our first “$class” structure we can see that the “NSMutableArray” inherits functionality from both “NSArray” and “NSObject” which may give us further  hints about what data might be held.

So, this “NSMutableArray” is going to contain a collection of other objects, and looking at the rest of this object’s structure we find a key “NS.Objects” which is followed by an array containing just that collection. This array has only one item, another object containing a “$class” definition, so let’s take a look. This time our “$classname” is particularly useful: “BubbleItem” – making reference to the “speech bubbles” displayed on screen; and indeed we find message details (author, message text, timestamp) in the object’s data fields.

There’s got to be an easier way…

NSKeyedArchiver files can contain really key evidence but even a data-sadist like me is going to lose their mind converting all of these files into a usable format by hand; so how can we speed things up?

Well our PIP tool has a “Unpack NSKeyedArchiver” feature which reveals the object structure so that you can use XPaths to parse the file (either by using one of many already in the included XPath library or by writing your own).

Also, if you are a Python fan (if not, why not?) I have updated our ccl_bplist python module, which you can get here, with a “deserialise_NsKeyedArchiver” function to unpack the “$top” object.

I hope you found this blog post useful. As always, if you have any questions or suggestions, leave a comment below or contact the R&D team at research@ccl-forensics.com.

Alex Caithness, Lazy Data-Sadist, CCL-Forensics

Special thanks to Arun Prasannan for assisting the BlogKeeper by rendering the code readable.

Free Python module for processing plist files – get ’em while they’re hot, they’re lovely!

As anyone who has examined an iOS device (or an OSX device for that matter) will know, property list files are a major source of potential evidence. Being one of the main data storage formats they might contain anything from configuration details to browsing history to chat logs.

We’ve examined property lists in detail previously, covering their file formats and the challenges they might present (you can download the white paper here). We have also released our tool PIP, which can be used to parse data from plists and other XML data in a structured way.

As regular readers might have gathered I’m pretty keen on the Python scripting language and while the built-in libraries do support reading from the XML property list format (available through the plistlib module) there is no support for the binary property list format which is increasingly becoming the standard format.

Recently I had a task where I needed to parse a large number of binary format property list files inside a script I was writing. It was theoretically possible to have the script export them all to a single location and then perhaps run PIP separately but I really wanted to have everything self-contained and besides, I needed to make use of the data extracted from the plist files later on in the script.

One of the beauties of Python is how quickly you can go from concept to “product”, so I decided that rather than waste time finding a work-around I would craft a proper solution. After a few hours’ work I had a fully functioning module for dealing with binary plist files in Python and today we’re excited to be releasing the code so that other practitioners and coders can make use of it.

You can get the source from our Google Code page (while you’re at it you can also download our module for dealing with BlackBerry IPD backup files here.

I designed the module to provide the parsed data in as native a format as possible (see Table 1) so when writing your code you do not have to deviate from the normal Pythonic constructs – the data structure returned contains everyday Python objects. The data structure returned is also vastly interchangeable with that returned by plistlib’s “readPlist” function (the exception being that plistlib wraps “data” fields in a “Data” object rather than giving direct access to the underlying “bytes” object).

Table 1: Converted data types returned by ccl_bplist

The module only has one function that you need to know about in order to use it: “load()”. This function takes a single argument which should be a binary file-like object* and it returns the python representation of the data in the property list.

In addition to the ccl_bplist module, the Google Code repository contains an example of the module in use, parsing the “IconState.plist” from an iOS device auditing the Apps and folders present on the Springboard home screen.

Icon state screenshot

Script results

We really hope that the community will be able to make use of this module and if you have any questions please leave a comment or email us at research@ccl-forensics.com.


*When working from a file on disk you should use the open() function with “b” in the mode string e.g.: open(file_name, “rb”).

There may be other times when the bplist has come from another source, e.g. a BLOB field in a database or even a property list embedded in another’s data field. In these cases you can wrap a bytes object in a BytesIO object found in the “io” module e.g.: io.BytesIO(some_data).