Chrome Session and Tabs Files (and the puzzle of the pickle)

In this blog post Alex Caithness investigates the file format and contents of Chrome’s “Current Tabs”, “Current Session”, “Last Tabs” and “Last Session” files and discovers that, even with the original source code at your side, you can still end up getting yourself into a Pickle.

A link to a Python script for automating the process can be found at the end of the post.

I’ve been on a bit of a browser artefacts kick as of late, digging around both on desktop and mobile platforms for stuff I haven’t tackled before. Taking a peek in my preferred browser’s (Chrome) “AppData” folder revealed that the ubiquitous-ness of SQLite as a storage format means that inspecting the data for a lot of artefacts has been made pretty simple. I had also recently tackled the Chromium web-cache format for another project (the format is now also used both on Android and RIM Playbooks) and, with the pain that caused me still fresh in my mind I had no desire to revisit it. There were, however, four likely looking candidates for a quick probing in the form of the “Current Tabs”, “Current Session”, “Last Tabs” and “Last Session” files.

Chrome AppData Folder

Hello there…

Broadly speaking, these files store the state of the opened tabs, their back-forward lists and the sites displayed therein. The files can be used by Chrome to restore your previous browsing session when you restart the browser (if that’s how you have it set up) or in the event of a crash. It turns out that these files can contain some really rich data, but first you had to do battle with the file format…

In previous posts I’ve made mention of the usefulness of having access to the source code that governs the format in which the data is to be stored, and as Chrome is open source I was heartened. “This shouldn’t be too tricky,” I thought to myself as I set about finding the ‘few lines of code’ which would unlock the file’s secrets… Let me tell you now: the Chrome source is a sprawling behemoth and my journey across the codebase (and on one occasion, outside of it) was long and arduous, and, when it comes down to it, it all boils down to understanding the ‘Pickle’…

Header of the Session file

The file header

The file header was easy to track down, I headed over to the definition for session_backend (src/chrome/browser/sessions/session_backend.cc) where we confirm that “SNSS” is simply a file signature followed by a 32bit integer giving the version of the file, which, at the time of writing, should always be 1 (all data is stored little-endian). Also in this file we encounter a method named “AppendCommandsToFile” which appears to be responsible for writing the details into the files. The method describes that for each record, a 16-bit integer is written to the file giving the size in bytes of the record (not including this value), followed by an 8-bit “id” (which appears to relate to the ‘type’ of the record) and the contents of the “SessionCommand”.

Record structure overview

Record structure overview

So now I knew what the overview of the structure in the file was: a nice simple size, contents, size, contents, size, contents… etc. file format, with the records written sequentially, one after another. But I still had no information about the structure of those contents. SessionBackend was operating with a SessionComand object so I tracked down the source code describing this object (src/chrome/browser/sessions/session_command.h) but was disappointed to find the following explanation in the source code’s comments:

“SessionCommand contains a command id and arbitrary chunk of data. The id and chunk of data are specific to the service creating them.”

OK, so the information I wanted isn’t going to be here, but the comments go on to say:

“Both TabRestoreService and SessionService use SessionCommands to represent state on disk”

Aha! So although I hadn’t quite found what I was looking for here, I have found a useful signpost pointing in the right direction. Now, neither “TabRestoreService“ (src/chrome/browser/sessions/tab_restore_service.h) nor “SessionService” (src/chrome/browser/sessions/session_service.h) themselves give us the information we’re after, but both of them ‘inherit’ from a common base class called “BaseSessionService” (src/chrome/browser/sessions/base_session_service.cc) (I gave a brief overview of object oriented principals including inheritance in a previous blog post)  and it is in BaseSessionService where we finally get what we’re after…

BaseSessionService contains a method called “CreateUpdateTabNavigationCommand” which is responsible for writing that “arbitrary chunk of data” into the SessionCommand which eventually gets written to disk. The record starts with a 32 bit integer which gives the length of the data (this is in addition to the length value outside the SessionCommand). The rest of the SessionCommand’s contents structure is described in the table below.

SessionCommand serialisation

SessionCommand structure

Data type Meaning
32 bit Integer Tab ID
32 bit Integer Index in this tab’s back-forward list
ASCII String (32 bit Integer giving the length of the string in characters followed by an ASCII string of that length) Page URL
UTF-16 String (32 bit Integer giving the length of the string in characters followed by a UTF-16 string of that length) Page Title
Byte string (32 bit Integer giving the length of the string in bytes followed by a byte string of that length) “State” (A data structure provided by the WebKit engine describing the current state of the page. We will look at it in detail later)
32 bit Integer Transition type (explained below)
32 bit Integer 1 if the page has POST data, otherwise 0
ASCII String (see above) Referrer URL
32 bit Integer Referrer’s Policy
ASCII String Original Request URL (for example if a redirect took place)
32 bit Integer 1 if the user-agent was overridden, otherwise 0

As SessionCommands contents can be populated by other means, not every Session command contains data formatted as shown above. During testing it was shown that it is the SessionCommand’s 8-bit ID which identifies whether the record contains this kind of data (when the ID was 1 or 6 then this data format was found). Those with other IDs were typically much shorter (usually around16-32 bytes in length) and did not appear to contain information which was of so much interest.

There are a few fields in the table above which are worth taking a closer look at; the “State” field we’ll explore in detail later as it’s a complicated one. The “Transition type” is a little easier to explain; this field tells Chrome how the page was arrived at. The field will be an integer number, the meaning of which is described in the tables below. The value is essentially split into two sections: the least significant 8-bits of the integer give a type of transition and the most-significant 24-bits form a bit-mask which gives other details. These details are gathered from page_transition_types (content/public/common/page_transition_types.h).

Least Significant 8-bits Value Meaning
0 User arrived at this page by clicking a link on another page
1 User typed URL into the Omnibar, or clicked a suggested URL in the Omnibar
2 User arrived at page through a  bookmark or similar (eg. “most visited” suggestions on a new tab)
3 Automatic navigation within a sub frame (eg an embedded ad)
4 Manual navigation in a sub frame
5 User selected suggestion from Omnibar (ie. typed part of an address or search term then selected a suggestion which was not a URL)
6 Start page (or specified as a command line argument)
7 User arrived at this page as a result of submitting a form
8 Page was reloaded; either by clicking the refresh button, hitting F5 or hitting enter in the address bar. Also given this transition type if the tab was opened as a result of restoring a previous session.
9 Generated as a result of a keyword search, not using the default search provider (for example using tab-to-search on Wikipedia). Additionally a transition of type 10 (see below) may also be generated for the url: http:// + keyword
10 See above
Bit mask Meaning
0x01000000 User used the back or forward buttons to arrive at this page
0x02000000 User used the address bar to trigger this navigation
0x04000000 User is navigating to the homepage
0x10000000 The beginning of a navigation chain
0x20000000 Last transition in a redirect chain
0x40000000 Transition was a client-side redirect (eg. caused by JavaScript or a meta-tag redirect)
0x80000000 Transition was a server-side redirect (ie a redirect specified in the HTTP response header)

NB during testing, although the transition types looked correct in the “Current Session” and “Last Session” files, in the “Current Tabs” and “Last Tabs” files the transition type was always recorded as type 8 (Reloaded page).

When it comes to the record structure, there is still a little more to the story, and yes, this is where the Pickles come in.

This data structure is not being written directly to a file, but rather to what Chrome calls a “Pickle” (src/base/pickle.h). A Pickle is a sort of ‘managed buffer’; a way for Chrome to write (and read) a bunch of values, like those in the tables above, into an area of memory in a controlled way. Indeed, the “length-value” structure we see with the strings is down to the way Pickles write strings into memory, as is the, apparently superfluous, extra ‘length’ field at the start of the record structure. One other pickle-related side-effect which isn’t necessarily immediately obvious when you look at the data in a hex editor is that pickles will always write data so it is uint32-aligned. This means that data will always occupy blocks of 4 bytes and if needed (such as in the case of strings) will be padded to ensure that the next data begins at the start of the next 4-byte block.

It turns out that the contents of the mysterious “State” field are also governed by a Pickle. This field contains serialised data from the WebKit engine. The data is held in a “NavigationEntry” (content/public/browser/navigation_entry.h) “content state” field, but is originally populated by glue_serialize  (webkit/glue/glue_serialize.cc). It duplicates some of the data that we have already described from the outer record, but also contains some more detailed information regarding the state of the page, not least the contents of any forms on the page. The code describing the serialisation process is found in glue_serialize in the WriteHistoryItem method.

The state byte string begins with a 32 bit Integer giving the length of the rest of the record (this is in addition to the length defined in the outer record structure) and then continues with the “WebHistoryItem” structure shown in the table below:

WebHistoryItem structure

WebHistoryItem structure

Data type Meaning
32 bit Integer Format Version
String (see below) Page URL
String (see below) Original URL (for example if a redirect took place)
String (see below) Page target
String (see below) Page parent
String (see below) Page title
String (see below) Page alternative title
Floating point number (see below) Last visited time
32 bit Integer X scroll offset
32 bit Integer Y scroll offset
32 bit Integer 1 if this is a target item otherwise 0
32 bit Integer Visit count
String (see below) Referrer URL
String Vector (see below) Document state (form data) – explained in more detail below
Floating point number (see below) Page scale factor (Only present if the version field is greater than or equal to 11)
64 bit Integer “Item sequence number” (Only present if the version field is greater than or equal to 9)
64 bit Integer “Document sequence number” (Only present if the version field is greater than or equal to 6)
32 bit Integer 1 if there is a “state object” otherwise 0 (Only present if the version field is greater than or equal to 7)
String (see below) “State Object” (only present if the value above is 1 and the version field is greater than or equal to 7)
Form data (see below) Form data
String (see below) HTTP content type
String (see below) Referrer URL (again, for backwards compatibility apparently)
32 bit Integer Number of sub-items in the field below
WebHistoryItem Vector (see below) A number of sub items (for example embedded frames). Each record has the same structure as this one

That table has a lot of “See below” in it, so let’s get down to explaining some of the subtleties/oddities that this data structure provides.

Strings: strings are actually stored differently to those in the outer record. Despite the fact that the data is still being written into a Pickle, the source code uses a different mechanism to do so. The source code forsakes the Pickle’s built in string serialisation methods (for reasons best known to the Chrome programmers), instead taking a more direct route of writing the length of the string directly, followed by the in-memory representation of the string. Basically, this results in the string fields comprising a 32-bit Integer giving the length of the string followed by a UTF-16 string only, this time the length refers to the length in bytes, not the length in characters. To further confuse matters, if the length is -1 (0xFFFFFFFF) this indicates that the string is not present (or ‘null’ in programming terms) or un-initialised (and therefore empty). There is an exception to this structure: if the version field is 2, where, as the comments in the source code suggest, the format was “broken” and stored the number of characters, this was fixed in version 3 onwards.

String Vector: “Vector” in this case essentially means ‘List’. The vector begins with a 32-bit Integer giving the number of entries in the list which is then followed by that many strings in the format described above. In the data structure above this is used to serialise what is described as the “document state”. In testing this appeared to contain information regarding any form fields that may be present on the page (including hidden fields). The list of strings can be broken up into groups of 3 strings, the first of which gives the name of the form field, the second the type of field and the third the current contents of the field.

Floating Point Numbers: IEEE 754 double-precision floating point numbers are used as a representation, but Pickles do not directly support this data type. Because of this, the code uses the Pickle’s “WriteData” method, passing the internal, in-memory representation of the floating point number into the Pickle. The upshot of using the “WriteData” method is that the 64-bit floating point number is prefaced with a 32-bit integer giving the length of the data (which will always be 8 for a double-precision float).

Form Data: the (slightly convoluted) format for this data serialisation is detailed in the WriteFormData method in glue_serialize, however across testing this data was never populated so I can’t vouch for its contents.

Sub items: this contains further WebHistoryItems for any embedded pages or resources on the page. During testing I saw it used to store details of adverts, Facebook “like” buttons and so on. The structure for these sub items is identical to the structure described in the table (note, however, that unlike the top-level WebHistoryItem they do not begin with a size value).

So that’s the structure of the file – not the most pleasant file format I’ve ever dealt with and, even with the source code on hand, it was a lengthy task. So was it worth it?

Well first the case against: a lot of the data is duplicated in other places, not least the History database (which is SQLite so much nicer to work with), and between the “Current” and “Last” versions of the files you only have information regarding 2 sessions worth of browsing, although, increasingly in today’s “always-on” culture, this could still account for a significant period of browsing. Which brings me to the other significant disappointment for these files – timestamps (or rather the apparent lack of them); of course, this makes perfect sense when you consider what Chrome needs the files for – timestamps simply aren’t required for restoring sessions, all the same, it’d make the file more useful to us if they were there.

But it’s not all doom and gloom (which is lucky, otherwise this blog post would be a bit of a waste of time). Firstly, although we only have 2 sessions worth of browsing live on the system, colleagues have already demonstrated to me that there is plenty of scope for recovering previous examples of the files – especially from volume shadow copies, and the 8-byte long static header means that carving files from unallocated space may be possible (no footer though, so some judgement would need to be made regarding the length of the files). Probably more importantly these files give us access to information which it would be tricky to acquire otherwise (or at the very least another opportunity to recover information which may have been deleted); the form contents are obviously a nice additional source of intelligence, both in terms of user credentials, email addresses and possibly message contents (I was able to recover Facebook chat messages from the form data in the “document state” for example). Also, the presence of the transition types, referrer and requested URL fields means that you can build up detailed browsing behaviour profiles, tracking the movement between sites and tabs.

This is not a file format that I would want to parse by hand again, so to automate the process I have written a Python script which we’re happy to make available to the forensics community. The script is designed both as a command line tool which generates a simple HTML report and a class library in case anyone wishes to integrate it into other tools (or create a different reporting format). You can download the script from http://code.google.com/p/ccl-ssns/.

As always, if you have any comments or questions you can get in touch in the comments or by emailing research@ccl-forensics.com

Alex Caithness

Android Ice Cream Sandwich Browser Cookies (and other artefacts)

The Android browser traditionally had data structures that were distinctly Android; but as Alex Caithness explains, there are signs of convergence with another of Google’s pet projects…

I should probably start by explaining that Android has a delightful habit of naming its operating systems after desserts. The upside of this is that it’s quirky; the downside is that cake consumption in the lab increases by a significant factor.

Hence the name “Ice Cream Sandwich”.

Across previous versions of Android, the cookie storage format has remained unchanged: they have been neatly stored in the browser’s “databases” folder in the “cookies” table of the “webview.db” SQLite database; this appears to have changed in version 4.0 of Android AKA Ice Cream Sandwich (ICS).

Firstly, what is peculiar is that the “webview.db” file still contains the legacy “cookies” table, however in testing this was never populated. Instead, a new database named “webviewCookiesChromium.db” is used to store cookie data.

The name of the file gives us a big clue to the nature of the file – we’re seeing a convergence between the Android browser and Chromium (the browser upon which Google Chrome is built). Investigating the database confirms this; the schema and structure of data in this new database is identical to that of Chrome’s.

The great news for Dunk! users is that they can go right ahead and use the Google Chrome decoder on this file to parse and extract the cookies held.

There is also a second cookies database present in ICS named “webviewCookiesChromiumPrivate.db”. This database contains cookies transmitted while an “Incognito Tab” (the private browsing feature) is being used. The structure is identical to the other database; however, when the incognito tab is closed the file is truncated to 0 bytes.*

Further evidence of this convergence towards Chrome comes from the cache structure which, like the cookies, has moved to the same structure as is found in Chrome. For more details, take a look at http://www.chromium.org/developers/design-documents/network-stack/disk-cache.

*Although further research is required we anticipate that epilog will be able to recover these records from a raw dump of the flash chip!

Alex Caithness

R&D Team

Dunk your cookies in our software

Cookies are often seen as the poor cousin of digital evidence, but they can provide a wealth of information for digital investigators – including how often, from where and how a user visited a certain site – as well as the search terms used to find it.

So how can we access this treasure trove of knowledge?

By using a piece of software called dunk! which covers all the main PC internet browsers (Chrome, Firefox, Safari, IE, etc.) and a wide range of mobile browsers.

The inspiration for dunk! came after conducting an examination of an iPhone during which we found that evidence for the web history and cache was thin on the ground – although we were getting some interesting key word hits in the cookies.

Previously, analysts had been dumping cookies into a straightforward table view, but not looking at the structure of the cookies’ values. However, in this case all the interesting key words fell inside what were found to be Google Analytics cookies. The nice thing about these cookies was that, unlike many cookies where the structure is proprietorial, these were consistent between all sites and contained really interesting insights into a user’s web activity.

We wrote a program enabling us to view all the cookies at once, and where known structures (such as Google Analytics) were found, automatically parse them – and we designed it to support as many browsers as possible.

But that’s not all it does; dunk! can detect session cookies which may contain usernames, email addresses, and sometimes even passwords, allowing investigators to build the fullest picture possible of browsing habits.

The interface allows the data to be filtered, searched and exported. In a nutshell, the software does the following:

  • Processes cookies from PCs and mobile devices
    • Internet Explorer 5+
    • Mozilla Firefox 3.x
    • Mozilla Firefox 4.0
    • Google Chrome
    • Safari browser
    • Opera 5+
    • Apple “binarycookies” format
    • Android browser
    • Flash cookies
    • Nokia 40 browser
  • Parses Google Analytics cookies
  • Parses Adobe Flash cookies
  • Enables investigators to search and filter evidence
  • Detects session cookies which may contain usernames, email addresses, etc.
  • Outputs to TSV and XML file formats

Open the cookie jar and take a detailed look at what’s inside.

Alex Caithness

Dunk! Developer

Updated signature files for epilog

CCL-Forensics’ developers are constantly adding new files to increase the capability of epilog, and the latest signature files are now available for download, free of charge.

These new signature files now contain support for Apple iOS5.

We also welcome suggestions for additions to future signature file releases. Please email us at epilog@ccl-forensics.com.

What is epilog?

For those who don’t know, epilog is a software tool which allows investigators to recover deleted data from the widely-used database format, SQLite. Take a look at the first of our epilog videos:

It was developed by CCL-Forensics and – put simply – it gives investigators access to more data which could prove crucial in an investigation.

Many devices (whether mobile phones, computers, satnavs or other devices) store data in the SQLite database format.

Data stored in this type of database can provide a huge evidential opportunity for investigators. Many “off-the-shelf” tools can be used to view the live records in the database, but epilog  extracts deleted and de-referenced data from the database files or across a disc image or hex dump.

epilog’s three recovery algorithms can be used on any SQLite database, regardless of the type of data stored. However, epilog signatures can be used to tailor its behaviour to a particular database.

Included with the initial release of epilog were signatures including:

  • Android (SMS, call logs, calendars, address book and others)
  • iPhone (SMS, emails, calendar, and others)
  • Smartphone third party applications (including Yahoo Messenger, eBuddy chat and others)
  • Safari (internet history and cache and others)
  • Mozilla (cookies, internet history, form data and others)
  • Chrome (internet history)