Boring, nerdy sail stuff: How to convert SAS Planet cache files

The Red Mosque of Colombo.  If you are not sailing nerd, this is the last interesting thing you will see in this post.  Run away now.

Your friendly cruise director, Jen, has taken the day off. Unfortunately for you, you're left with Matt to fill the blog void.  And this week, instead of a relentless recap of our boring lives (especially here and now, stuck in Galle), 99.9% of you are going to get treated to something you've probably never wanted: A tutorial on how to convert cache files in SAS Planet. If you've never heard of SAS Planet before, this is probably your cue to shake your head, close the browser, and try your luck again in a week or two. If however, for the remaining 2 or 3 of you, you DO know about SAS Planet, then stick around.

UPDATED Dec 1, 2020- Check the bottom for another option in addition to what's detailed below.

 

Why you might want to convert your cache structure:

SAS Planet ("SAS") is a wonderful tool for offline map and imagery viewing. It allows for virtually unlimited storage of all sorts of map types (Google Satellite, Navionics, Bing, ARC.GIS, and many others).  It also allows for integration with a GPS feed to show current position, speed and heading overlaid on any of the stored maps. 

However, as the number of your stored maps grows, the storage container for the maps, or "cache", becomes unwieldy and difficult to move, copy, or share with others. This is because the original cache structure (assuming you are using the standard "SAS.Planet" cache type) was set up on a series of nested folders to organize and contain the literal millions of small images that are stitched together to create the maps shown on the computer. Each image file is only 1-2 Kb, but a good sized SAS cache might contain 10 million of these small files. The problem arises when you go to backup or copy your cache. Computers are becoming faster and faster with passing data, but the problem here lies in the overhead associated with the Windows (or any other) operating system. 

When copying a file, you, the user, only worry about dropping and dragging, or hitting Ctrl-C then Ctrl-V. However behind the scenes the operating system is having this sort of conversation:

OS: Hey, Disk, User wants to copy file x to location y.

Disk: OK.

OS: Does that location exist?

Disk: Nope

OS: OK, create it.

Disk: OK, created.

OS: Are you ready to receive the data?

Disk: Yes

OS: Here's the data [this is where the 1.8 KB gets sent along]

Disk: OK, got it.

OS: Did you get all of it?

Disk: Hold on, let me check... Yep, it's here.

OS: OK, that's it. Goodbye.

When copying a large file, that "conversation" overhead  is almost unnoticeable.  However, when copying millions of small files, that overhead is where the computer spends most of its time relative to the small image file that it passes along. Consequently, copying or backing up a 10 GB cache might take hours, whereas a 10 GB movie file might only take a minute or two.

Luckily, more recent versions of SAS have provided support for a different cache format known as "SQLite3". Instead of storing millions of individual image files, the cache has a limited nested directory structure which ends in one or more small database files. Each database can contain thousands of images.

Let's use my Google Satellite cache as an example. When converted to a SQLite3 format, the cache went from 3,406,505 Files and 114,444 Folders to 1,681 Files and 992 Folders. That is a whopping 99.96% reduction in the number of files!  This allows the operating system to copy and back up the cache files much more efficiently.

How to convert your cache:

So now that you understand the need to convert the cache, let's get in to how to actually get it done. Within SAS, there is Cache Conversion utility.  However, it's rather manual and takes a bit of setup to get it to work properly. That said, it's not terribly complicated and the payoffs are large, so it's worth doing.

The first thing to understand is that the SAS Cache Conversion utility only allows you to convert one map source at a time. So, depending on the number of map types in your cache, you may need to repeat this process several times. Also, to prevent confusion, it's best not to have SAS attempt to download more new files while you're are doing this process, so for now set your data source to "Cache Only".

Step 1- Navigate to the main SAS directory.  This is easily found by opening Windows File Explorer and navigating to your SAS Planet folder.  In this main directory you should see a sub-directory called "cache". This is where your existing cache files reside. Create a new directory called "cache_sqlite". At this point you should now have a "cache" sub-directory and a " cache_sqlite" sub-directory. You only need to create this "cache_sqlite" directory once.  This is the new directory where all of our new SQLite3 map specific caches will reside. 

Proper set up of the directory structures is the key to a successful conversion

 

Step 2- Take a look in your "cache" folder   Within "cache" you should see many subfolders with names like "sat", "both", "navionics", "vesat", etc. Each one of these sub folders represents a different map type that you have downloaded at some point.  Most names are pretty self explanatory, but a few are more cryptic.  Here are a few you may not immediately recognize:

•  "sat" --> Google satellite imagery
• "Both" --> Google Hybrid images (this is the overlay that shows town names, streets, businesses, etc.)
• "vesat" --> Bing satellite imagery

Determine which map type you want to convert. Let's use the Google satellite imagery as an example. Drill down into the sub-directories under that map type ("sat") until you reach the final level that contains the image files. Make a note the type of image file that the map type uses (possible to be .jpg, .png, .bmp or even others). Google satellite imagery uses .jpg. Come back up to main SAS directory.

Step 3 - We now need to make a new sub-folder in our "cache_sqlite" directory  that is the EXACT same name as the sub-folder in the "cache" directory that we want to convert. So if we are going to convert Google Satellite imagery, we need to create a "sat" sub-folder within our "cache_sqlite" folder.  The sub-folder names must be identical or the SAS program won't recognize the new location.

This completes the "set up" required, and we are now ready to start the SAS Cache Conversion utility.

Conversion utility screen

 

 Step 4 - Within the SAS program, navigate to Operations -> Cache Manager

This will bring up a bring up a window where we need to enter some info for the conversion to work properly.

·         In "Source Cache", leave the first drop-down set to "Folder".

·         In "Path", click on the ellipses button to the right of the blank field and navigate to and select the source folder, namely your "sat" subfolder within the main cache folder.

·         Set the "Format" drop down to "Sas.Planet"

·         Set the "Extension" dropdown to ".jpg" (this is the file type you discovered when you drilled down into the cache in Step 2 of the set up process)

·         Ignore all the other fields in the top half, or "Source Cache" area.

·         In the "Dest Cache"", leave the first drop-down set to "Folder".

·         In "Path", click on the ellipses button to the right of the blank field and navigate to and select the destination folder, namely your NEW "sat" subfolder within the newly created "cache_sqlite" folder.

·         Set the "Format" drop down to "SQLite3"

·         Ignore all the other fields in the bottom half, or "Dest Cache" area.

You are now ready  to run the conversion. Click the "Start" button. A progress window will pop up. It will take a bit of time for the system  to determine what it needs to process, but after a few minutes (possibly longer if your cache is really big), the progress window will start showing how many files it has processed.  Just for planning purposes, a large cache of roughly 6 million files (58 Gb) took almost 13 hours to process. However this was on a machine limited to USB2 and the cache was on an external disk. I would assume that the ability to use USB3 would have sped things up considerably.

 

Setting up SAS to use the new SQLite cache type.

Step 5 - Once the process is complete, if you want to check that everything went smoothly, you need to tell SAS to start using the new cache. This can be done as follows:

·         Navigate within the SAS program to Settings -> Options

·         Select the "Cache" tab

·         Make sure the "SQLite3 cache folder" is set to "cache_sqlite"

·         Change the "Default cache type" dropdown to "SQLite3"

·         Click "Apply"

You should now be able to see your newly created cache.  At this point you should ONLY be able to see the one mapset you just converted (make sure the selected basemap is properly set).  All the other map types will be grey, as that data has not been converted yet. 

Step 6 - To finish the conversion for other map types, simply repeat steps 2-4 as needed.

Phew! Did you really read all that anyway? If so, I gotta suggest you find yourself a hobby.

 

UPDATE Dec 1, 2020:

Another option for this process is to use the program called "Sat2Chart" developed by Paul Higgins.  The program is primarily used to create .kap and mbtile charts from imagery sources like Google earth and Sas Planet.  However, Paul has also recently included a utility that automates the cache conversion process detailed in my post.  The utility is called "Convert_Cache" and is found within the "Sat2Chart" program.  You can download the program here: http://gdayii.ca/Downloads/  

To run Sat2Chart, you will also need to download and install OoRexx 5.0.0, which is also available from the download location given above.

When the Convert_Cache window opens, there is a "Tutorial" button you can click  that walks you through the operation and setup of the utility.