Tagged Pointer NSStrings

31 Jul

Mike Ash’s Friday Q&A this week talks about Tagged Pointers and NSString.

If we’re optimizing for ASCII, we might as well drop full Unicode support altogether. Strings containing non-ASCII characters can use real objects, after all. ASCII is a seven-bit encoding, so what if we allot only 7 bits per character? That lets us store up to eight ASCII characters in the 60 bits available, plus 4 bits left over for the length. This is starting to sound useful. There are probably a lot of strings in an app which are pure ASCII and contain eight characters or fewer.

AutoAssets build script

22 Jul

Working with Xcode image assets in code is pretty easy. Just include the images in a Xcode assets file and throw in some UIImage calls like so:

UIImage(named:"someImage")

Although it’s unlikely to cause any crashers, you can easily end up having a typo somewhere and ship your app with a missing image. Not to mention you have to remember and type out the image name every time you want to use it.

There are better ways of course. For example, you can make it a bit better by using string constants. You’ll end up with something like this:

UIImage(named: LAMSomeImage)

Or you could wrap them in a category

LAMAssets.SomeImage()

But Jesse Squires proposes a better way using enums, which also provides proper name spacing. You should really read his post, it has some great stuff on dealing with colors too.

With Jesse’s method you access the images using the following style

Assets.SomeImage.image()

The difference between his enum style and using a category is subtle in use, but significant in implementation.

Inspired by his post, but not wanting to keep enums up-to-date with my Xcode Assets, I made AutoAssets.

AutoAssets (written in Swift) is a build script that generates a Swift source file at build time based on the projects assets. Basically it takes Jesse’s enum idea and automates it.

Here’s the repo

NSData+Compression and API costs

21 Jul

During my Tinkering with CompressionLib series I built a small NSData extension (or category depending on your language of choice) to make compressing / decompressing data easier.

It had only two public methods:

func compressedDataUsingCompression(compression: Compression) -> NSData?

func uncompressedDataUsingCompression(compression: Compression) -> NSData?

I debated about adding initializers to the extension, but decided to hold off initially until I could give it some more thought.

It sounds trivial, but every bit of public API you expose is that much more you need to support. The cost of adding API isn’t really the code itself. Code is just one part. You have to consider the design, testing, documentation, maintainability, and support costs too.

What finally pushed me over the edge was reviewing NSData’s API.

NSData exposes convenience methods for initializing from the contents of a file

init?(contentsOfFile path: String)

And frequently when you’re dealing with compressed data it’s stored in a file on disk. So it seems only natural that you would expect to be able to both read it in and have it decompressed at the same time. After all, it is pretty unlikely that you’re going to read some compressed data into memory and leave it compressed.

So I added two convenience initializers to my NSData extension today.

convenience init?(contentsOfArchive path: String, usedCompression: Compression?)

Which returns a NSData object initialized by decompressing the data from the file specified by path using the given compression algorithm.

And the shorter “smarter” version

convenience init?(contentsOfArchive path: String)

Which returns a NSData object initialized by decompressing the data from the file specified by path. Attempts to determine the appropriate decompression algorithm using the path’s extension.

This method is equivalent to NSData(contentsOfArchive:usedCompression:) with nil compression

Here’s the repo for the Swift version.

Or, if you prefer, you can grab the Obj-C version here.

Tinkering with CompressionLib (Part 3)

18 Jul

Continuing my series on Tinkering with CompressionLib; if you haven’t read my previous posts you can check them out here – Part 1 and Part 2.

Last time when working with CompressionLib I introduced a category on NSData to allow for easy compression / decompression. I initially wrote this in Objective-C.

While working with C libraries in Swift is fairly easy, it’s still nowhere near as easy as dealing with them in Objective-C; largely since you can mix C and Objective-C in the same source file and the fact that C couldn’t hardly care less about type safety.

This time we’re going to take a look at working with CompressionLib from Swift.

// alloc some memory for the stream
let streamPtr = UnsafeMutablePointer<compression_stream>.alloc(1)
var stream    = streamPtr.memory

// the stream’s status
var status : compression_status

// we want to compress so we use the ENCODE option
let op = COMPRESSION_STREAM_ENCODE

// COMPRESSION_STREAM_FINALIZE is used to indicate that no further input will be added. Since we have the entire input data we can finalize right away
// compression_stream_process() expects flags to be an Int32, so we need to convert our flag to that type
let flags = Int32(COMPRESSION_STREAM_FINALIZE.rawValue)

// we want to use the super awesome LZFSE algorithm
let algorithm = COMPRESSION_LZFSE

// init the stream for compression
status = compression_stream_init(&stream, op, algorithm)
guard status != COMPRESSION_STATUS_ERROR else {
    // FIXME: Shame on you for not handling this error properly
    return
}

// setup the stream's source
let inputData   = // get some data
stream.src_ptr  = UnsafePointer<UInt8>(inputData.bytes)
stream.src_size = inputData.length

// setup the stream's output buffer
// we use a temporary buffer to store the data as it's compressed
let dstBufferSize : size_t = 4096
let dstBufferPtr  = UnsafeMutablePointer<UInt8>.alloc(dstBufferSize)
stream.dst_ptr    = dstBufferPtr
stream.dst_size   = dstBufferSize
// and we store the output in a mutable data object
let outputData = NSMutableData()


repeat {
    // try to compress some data
    status = compression_stream_process(&stream, flags)
    
    switch status.rawValue {
    case COMPRESSION_STATUS_OK.rawValue:
        // Going to call _process at least once more
        if stream.dst_size == 0 {
            // Output buffer full...
            
            // Write out to mutableData
            outputData.appendBytes(dstBufferPtr, length: dstBufferSize)
            
            // Re-use dstBuffer
            stream.dst_ptr = dstBufferPtr
            stream.dst_size = dstBufferSize
        }
        
    case COMPRESSION_STATUS_END.rawValue:
        // We are done, just write out the output buffer if there's anything in it
        if stream.dst_ptr > dstBufferPtr {
            outputData.appendBytes(dstBufferPtr, length: stream.dst_ptr - dstBufferPtr)
        }
        
    case COMPRESSION_STATUS_ERROR.rawValue:
        // FIXME: Eat your vegetables; handle your errors.
        return
        
    default:
        break
    }
    
} while status == COMPRESSION_STATUS_OK

// We’re done with the stream - so free it
compression_stream_destroy(&stream)

// Finally we get our compressed data
let compressedData = outputData.copy()

And just like before, when you want to decompress the data you just need to change two variables.

// set the stream operation to decode instead of encode
let op = COMPRESSION_STREAM_ENCODE

// set the flags to 0, we don't need any flags here
let flags = 0

Correction, Stephen Canon points out that you actually can import CompressionLib directly; no need for a bridging header or module work around.

Just use import Compression

Thanks Stephen!

Now there’s one small catch. To be able to work with CompressionLib from Swift you have two options.

The simple way:

#import <compression.h> in an Objective-C bridging header.

The module way:

You can use a module to provide an interface between Swift and the C library. To do this first create a folder in your Project directory. You can name it whatever you like; I named it “CompressionLibModule”.

Next create a text file in the new directory and name it “module.map”

Add the following to the text file

module libcompression [system] {
    header "/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/compression.h"
    export *
}

This creates a module named “libcompression” that contains the compression.h header.

There’s one last step. In the Build Settings for your Project (or Target) under Swift Compiler - Search Paths you need to add the path to your module’s directory to the Import Paths

$(SRCROOT)/CompressionLibModule/

Now in any Swift file you want to use CompressionLib just import libcompression

The module option is quite a bit more work, and I’m sure this will get easier in the future.

So to wrap things up – I’ve rewritten the compression extension in Swift (don’t worry the Objective-C version is still available if you want it).

Here are the Swift versions of the NSData category’s public methods

// Returns a NSData object created by compressing the receiver using the given compression algorithm.
func compressedDataUsingCompression(compression: Compression) -> NSData?

// Returns a NSData object by uncompressing the receiver using the given compression algorithm.
func uncompressedDataUsingCompression(compression: Compression) -> NSData?

To compress some data you just call:

let compressedData = someUncompressedData.compressedDataUsingCompression(Compression.LZFSE)

And to uncompress some data:

let uncompressedData = someCompressedData.uncompressedDataUsingCompression(Compression.LZFSE)

Here’s the repo

 

Tinkering with CompressionLib (Part 2)

16 Jul

After thinking (and writing) a bit about the new CompressionLib available in OS 10.11 (and iOS 9) I’ve continued to play around with it a bit more.

Last time I had looked at the _buffer functions and how they could be used. I remarked, somewhat off handedly, about the inability to get at the expected uncompressed size of an archive and it being a limiting factor. The _buffer functions are simply not meant to be used that way. I do however, still yearn for a CompressionLib style way to get at the archives metadata (assuming the given archive format supports it of course) – I’ll have to give this some more thought.

I largely ignored the stream functions in my first post. This time around I’m going to take a much closer look at them and show how they can be used with NSData. Objective-C is getting some love today.

So, without further ado, let’s dive in.

Say you have an NSData object that contains some raw data, maybe level data for a game, or some other model data that doesn’t change frequently, and you would like to compress it. This is about the simplest case of using compression, but it’s also quite possibly the most common.

Here’s how we can use CompressionLib to compress a NSData object:

// the stream used for compression and it's status
compression_stream stream;
compression_status status;

// we want to compress so we use the ENCODE option
compression_stream_operation op = COMPRESSION_STREAM_ENCODE;

// COMPRESSION_STREAM_FINALIZE is used to indicate that no further input will be added. Since we have the entire input data we can finalize right away
compression_stream_flags flags = COMPRESSION_STREAM_FINALIZE;

// we want to use the super awesome LZFSE algorithm
compression_algorithm algorithm = COMPRESSION_LZFSE;

// init the stream for compression
status = compression_stream_init(&stream, op, algorithm);
if (status == COMPRESSION_STATUS_ERROR) {
    // FIXME: Shame on you for not handling this error properly
}

// setup the stream's source
NSData *inputData = // get some data
stream.src_ptr    = inputData.bytes;
stream.src_size   = inputData.length;

// setup the stream's output buffer
// we use a temporary buffer to store data as it's compressed
size_t   dstBufferSize = 4096;
uint8_t *dstBuffer     = malloc(dstBufferSize);
stream.dst_ptr         = dstBuffer;
stream.dst_size        = dstBufferSize;
// and we store the aggregated output in a mutable data object
NSMutableData *outputData = [NSMutableData new];

do {
    // try to compress some data
    status = compression_stream_process(&stream, flags);
    
    switch (status) {
        case COMPRESSION_STATUS_OK:
            // Going to call _process at least once more
            if (stream.dst_size == 0) {
                // Output buffer is full...
                
                // Write out to outputData
                [outputData appendBytes:dstBuffer length:dstBufferSize];
                
                // Re-use dstBuffer
                stream.dst_ptr = dstBuffer;
                stream.dst_size = dstBufferSize;
            }
            break;
            
        case COMPRESSION_STATUS_END:
            // We are done, just write out the dstBuffer if there's anything in it
            if (stream.dst_ptr > dstBuffer) {
                [outputData appendBytes:dstBuffer length:stream.dst_ptr - dstBuffer];
            }
            break;
            
        case COMPRESSION_STATUS_ERROR:
            // FIXME: Eat your vegetables, handle your errors.
            break;
            
        default:
            break;
    }
} while (status == COMPRESSION_STATUS_OK);

// We're done with the stream so free it
compression_stream_destroy(&stream);

// Finally we get our compressed data
NSData *compressedData = [outputData copy];

When you want to decompress the data you only need to change two variables:

// set the stream operation to decode instead of encode
compression_stream_operation op = COMPRESSION_STREAM_DECODE;

// set the flags to 0, we don't need any flags here
compression_stream_flags flags = 0;

The work involved in setting up a stream for compression / decompression is as minimal as it can be. Like the _buffer APIs, the _stream APIs are very clean and easy to use. In fact, the only stumbling block I ran into was the need to reset the dst_ptr and dst_size during processing to reuse the buffer. I had expected the reuse to be implied and didn’t see anything in the header about this.

With CompressionLib coming standard in the latest SDKs it seems only natural that NSData should support compression and decompression. Compressing some of your app’s data can go a long way to saving valuable space on the user’s device. Not to mention the performance and energy benefits of using compressed data during network transactions. So while I was playing with CompressionLib thats exactly what I did…

Here’s a small category on NSData that adds two public methods

// Returns a NSData object created by compressing the receiver using the given compression algorithm.
- (NSData *)lam_compressedDataUsingCompression:(LAMCompression)compression;

// Returns a NSData object by uncompressing the receiver using the given compression algorithm.
- (NSData *)lam_uncompressedDataUsingCompression:(LAMCompression)compression;

To compress some data you just call:

NSData *compressedData = [someUncompressedData lam_compressedDataUsingCompression:LAMCompressionLZFSE];

And to uncompress some data:

NSData *uncompressedData = [someCompressedData lam_uncompressedDataUsingCompression:LAMCompressionLZFSE];

Here’s the repo.

 

Tinkering with CompressionLib (Part 1)

13 Jul

There is a new system library in OS X 10.11 – CompressionLib.

For the compression algorithm itself CompressionLib exposes 5 choices.

COMPRESSION_LZ4
COMPRESSION_ZLIB
COMPRESSION_LZMA
COMPRESSION_LZ4_RAW
COMPRESSION_LZFSE // Apple specific

lzfse is the interesting one here. It’s an Apple developed algorithm that is faster than, and generally compresses better than zlib (the previous defacto standard). Since it’s Apple specific if you need cross platform support it isn’t for you. For now at least. Nothing has been announced, but I’m very hopeful Apple will choose to open source it. I think it’s in Apple’s best interest to get lzfse used as widely as possible.

CompressionLib’s public interface is small, but nicely thought out. It basically breaks down into two ways of dealing with compression, buffer based and stream based. There are, quite literally, only 7 functions:

compression_encode_buffer
compression_decode_buffer
compression_encode_scratch_buffer_size
compression_decode_scratch_buffer_size

compression_stream_init
compression_stream_process
compression_stream_destory

You can safely ignore the compression_encode_scratch_buffer_size and compression_decode_scratch_buffer_size functions. CompressionLib will automatically create the scratch buffer on your behalf if you pass a NULL scratch buffer to the encode / decode buffer functions. That knocks it down to just 2 functions for buffer based or 3 for stream based.

While watching WWDC ’15 – Session 712 “Low Energy High Performance: Compression and Accelerate” I decided to play around with the buffer functions.

size_t compression_encode_buffer(*dst_buffer, dst_size,
                                 *src_buffer, src_size,
                                 *scratch_buffer,
                                 algorithm)

The encode function compression_encode_buffer, and it’s decode counter part, compression_decode_buffer, take the same parameters and do exactly what you’d expect. You have to specify the dst_buffer‘s size (dst_size). During compression this isn’t a real limitation as you can reasonably expect the worst case scenario to be that the dst_buffer is the same size as the src_buffer (plus a few bytes for overhead when dealing with a very small piece of data). I really like this interface. It helps to make the memory management very clear. You create the buffers, you own them, and it’s your responsibility to free them. You can be reasonably sure of that just by looking at the function prototypes. The argument names also make it very clear exactly what each function needs. No need for a lot of documentation and reading here.

What about decoding?

size_t compression_decode_buffer(*dst_buffer, dst_size,
                                 *src_buffer, src_size,
                                 *scratch_buffer,
                                 algorithm)

This brings me to, what I believe is, a pretty big limitation. When using compression_decode_buffer you don’t have any reasonable expectation of what the uncompressed (dst_buffer) size should be. And if dst_buffer is too small compression_decode_buffer will simply truncate the result to dst_size. This is friendly, at least you don’t have to worry about a buffer overflow. But you must know beyond a reasonable doubt, exactly what size your uncompressed data will be. I halfway expected compression_decode_buffer to return the full size of the uncompressed data so you could increase the dst_buffer size and retry if needed. That would be wasteful, and it doesn’t. compression_decode_buffer returns the size of the data written to the buffer; if it’s truncated it simply returns dst_size. This is pretty clear in the header

@return
The number of bytes written to the destination buffer if the 
input is successfully decompressed. If there is not enough 
space in the destination buffer to hold the entire expanded 
output, only the first dst_size bytes will be written to the 
buffer and dst_size is returned.

I did some digging and couldn’t find any way of getting the expected uncompressed size. I then thought about inspecting the lzfse archive header, but there isn’t any published header spec (at least that I could find – if I’m wrong please let me know @leemorgan).

I think it would be highly beneficial if an API was provided to determine the expected uncompressed buffer size. Perhaps a function like the following:

extern size_t compression_decode_uncompressed_size(const uint8_t * __restrict src_buffer, size_t src_size, compression_algorithm algorithm);

I’ve filed an enhancement request asking for the ability to get the expected uncompressed data’s size. rdar://21787153

With that said, I need to thank Stephen Canon. We had a short conversation on twitter earlier today about CompressionLib and this “limitation”. While playing with CompressionLib I had been focused on lzfse and had neglected to think about how the library needs to provide support for other compression algorithms as well. I had (perhaps incorrectly) assumed that lzfse had this kind of metadata readily available in the archiver’s header. But failed to consider how it would work for archives that don’t have expected uncompressed size stored internally. Stephen pointed out that the _buffer interfaces are very low-level building blocks, and that it’s assumed that the higher level callers will keep this metadata around. The problem with this though, is that the higher level callers will implement this differently (and likely incompatibly). I might write one archiver that stores this data as extended attributes, while someone else might chose to wrap the archive itself with their own header.

But Stephen raises many good points. In lieu of the ability to get at the uncompressed data size directly, I think a standard way of accessing any available headers would be beneficial. This would keep the burden of dealing with the sizes at the higher level caller, while at the same time providing the higher level caller a standard way to get at any headers they might know about and be able to use (such as the uncompressed size).

I readily admit I might be making a bigger deal of this than it needs to be. One could always fall back to the stream functions. But the _buffer functions just look so damn beautiful, I want to use them.

Alara – Accessibility Enabled

10 Jul

Alara 1.1 is now available on the AppStore. I’m proud to say that Alara now fully supports VoiceOver. Accessibility is something I really wanted to get in 1.0, but the schedule just didn’t permit it.

So while 1.0 was waiting in the review queue, I got to work studying how to best bring accessibility support to Alara.

I have to admit something that is a bit shameful of me…I’ve never shipped an app with true accessibility support before. Sure, all apps get some accessibility support for free (big kudos to the Accessibility Team for their hard work there). But to truly be a good citizen, you need to take some time to add specific support.

Which of the Accessibility features you need to support largely depends on the style and usage of your app. If your app is primarily visual, as is the case with Alara, you’ll want to support VoiceOver. This will help users with vision impairments use your app.

When it comes to VoiceOver, the Accessibility Team has really done a lot of the leg work for us. You’ll go a long way by just giving some meaningful labels and traits to your views. You do this either by using the Accessibility Inspection in Interface Builder, or in code by setting the accessibilityLabelaccessibilityValue, accessibilityFrameaccessibilityTraits, and other properties as needed.

The work really comes down to truly understanding how your app works, how it can covey it’s information clearly and concisely, and then expressing that in a style that is more auditory than visual.

Take a look at this screenshot

alaraExample

If you don’t do anything specific to support VoiceOver the app isn’t very helpful. For example, if you were to tap on a bar in the bar-graph you wouldn’t get any information at all. You could tap on the UV index label in the bar (it’ll just read the number aloud), or the time label below it (hearing the time isn’t very helpful without any details). That information is separated and it isn’t very clear as you’re tapping or swiping around what you’re getting at.

In this case it makes more sense to combine these views in the “eyes” of VoiceOver so that they are seen as one entity and when selected reads something aloud like “UV Index 5 at 4PM”.

alaraExampleVoiceOver

I’m not certain I’ve got the wording just right yet and I’ll continue working to make it better as I go. If you’re a user of VoiceOver, or other accessibility features, and have recommendations please let me know either on twitter @leemorgan or email lee@shiftybit.net

The hardest part for me with accessibility was knowing where to get started. I’ll give you a hint. Turn on VoiceOver, launch your app, and start tapping around. You’ll get an idea about what needs a better description pretty quickly. Switch over to Xcode and give them whatever labels you think makes sense right then. Now go back and play with your app for a bit longer. You’ll probably notice all sorts of little things that don’t feel quite right. Maybe some information contained in two views is better understood by combining their description together? Maybe a button shouldn’t be VoiceOver enabled, and instead it’s parent view should be? These are all things you’re likely to think about once you start using your own app with VoiceOver. What are you waiting for? Go ahead, turn it on and play, then get to work. You’re users will thank you.